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EXECUTIVE  SUMMARY 


In  April  1978,  the  Federal  Aviation  Administration  (FAA)  conducted  an  experi- 
ment to  determine  the  effectiveness  and  acceptability  of  a Voice  Response 
System  (VRS)  designed  to  provide  pilots  with  a limited  preflight  weather 
briefing  via  touch-tone  telephone  without  the  intervention  or  assistance  of  a 
weather  briefer.  The  prototype  model  provided  three  weather  products:  hourly 
surface  observations,  terminal  forecasts,  and  grid  winds  aloft.  The  system 
communicates  by  voice  in  one  direction — from  computer  to  pilot. 

The  VRS  enables  pilots  to  communicate  with  a computer  to  obtain  weather  and 
flight  data.  The  pilot  enters  requests  for  specific  weather  reports  for 
locations  of  interest  by  depressing  the  keys  on  a touch-tone  telephone  or  key 
pad  adapter  on  non-touch-tone  telephones.  The  computer  will  "listen"  to 
telephone  signaling  tones  and  will  respond  with  a spoken  message  derived  from 
stored  human  speech. 

The  VRS  evolved  into  its  present  form  after  a period  of  controlled  experimen- 
tation by  FAA  personnel  who  sought  to  develop  suitable  procedures  and  to 
acquire  necessary  weather  products.  But  realistic,  live  experimentation 
was  necessary  to  find  the  ways  in  which  to  maximize  system  usefulness  and 
convenience  of  use.  Thus,  the  VRS  was  publicly  tested  in  the  greater  Washing- 
ton D.C.  area.  Questionnaires  were  mailed  to  a random  sample  of  1,778  pilots 
to  obtain  public  reaction  to  the  VRS.  Several  followup  efforts  were  made  to 
secure  information  from  nonrespondents. 

CONCLUSION'S . 

From  the  analysis  of  the  questionnaire  returns  and  telephone  contacts  with 
randomly  selected  nonrespondents,  it  is  concluded  that: 

l.  The  VRS  is  a practical  and  effective  means  by  which  pilots  can,  in  many 
instances , obtain  a satisfactory  overview  of  weather  conditions.  For  pre- 
flight planning,  56  percent  of  the  pilots  indicated  that  it  was  unnecessary 
to  call  the  Flight  Service  Station  (FSS)  for  weather  information  after  re- 
ceiving the  VRS  briefing  (figure  ES-1).  Of  those  who  still  found  it  neces- 
sary to  call  the  FSS  for  weather  information,  90  percent  indicated  that  for 
preflight  planning  the  VRS  briefing  reduced  the  time  on  line  with  the  FSS 
(figure  ES-2) . The  average  estimated  reduction  in  time  on  line  with  the  FSS 
for  preflight  planning  was  47  percent  (figure  F.S-3) . 

. The  VRS  opt  rating  procedures  are  simple  enough  for  most  pilots  to 
master  with  minimal  practice. 

V.  The  public  1'eaction  to  the  VRS  was  highly  favorable . Arialysis  of  the 
Sita  obtained  from  the  questionnaires  indicates  a high  level  of  acceptance 
of  the  VRS.  As  shown  in  (figure  ES-4)  93  percent  of  the  respondents  indi- 
cated that  the  VRS  is  either  "suitable  with  minor  changes"  or  "suitable  as 
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FIGURE  ES-4.  SUITABILITY  OF  THE  VRS  FOR  WIDE-SPREAD  PUBLIC  USE 


4.  No  significant  difference  in  the  results  existed  between  license  types 
or  experience  levels  of  general  aviation  pilots. 

b.  The  quality  of  speech  (i.e.,  pleasantness , rate , rhythm  t intonation , and 
intelligibility)  are  acceptable  for  public  dissemination. 

6.  Additional  weather  products  and  the  provision  of  NOT  AM's  would  provide  for 
better  overview  of  weather  and  flying  conditions. 


RECOMMENDATIONS. 

1.  The  following  seven  additional  products  should  be  incorporated  into  the 
Voice  Response  System  as  they  become  available. 

1.  PIREP's — Pilot  reports  on  weather 

2.  NOTAM's — Notices  to  airmen 

3.  SIGMET's — Significant  weather  information 

4.  AIRMET's — Airmens  Meteorological  Information 

5.  Weather  Synopsis 

6.  Route  Forecast 

7.  Area  Forecast 

spoken-name  readback  method  should  be  used  for  location  indent  if iers . 

impact  of  the  Voice  Response  System  on  FSS  specialist  workload  was 
connot  be  determined  in  the  Washington  metropolitan  area  due  to  many 
improvements  such  as  MAPS,  improved  PATWAS , TEL-TWEB,  800  lines  and  consoli- 
dation. In  the  event  a second  VRS  installation  is  implemented,  it  is  recom- 
mended that  a further  study  be  conducted  to  determine  the  effects  of  the  VRS 
on  specialists'  workload  and  that  this  study  be  performed  in  a controlled  field 
environment . 


2.  The 

3.  The 
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INTRODUCTION 


PURPOSE. 

The  purpose  of  this  report  is  to  document  the  results  of  the  Voice  Response 
System  survey  (VRS) . The  VRS  survey  was  conducted  to  determine  the  effectiveness 
and  acceptability  of  a computer-generated  voice  system  designed  to  provide 
pilots  with  a limited  preflight  weather  briefing  via  touchtone  telephone  with- 
out the  intervention  or  assistance  of  a weather  briefer. 

BACKGROUND. 

The  Federal  Aviation  Administration  (FAA)  plans  to  implement  a national  Flight 
Service  Station  (FSS)  automation  program.  This  program  will  enable  the  increas- 
ing demand  for  services  to  the  general  aviation  fleet  to  be  met  without  a large 
increase  in  operating  costs.  Increased  system  capacity,  easier  access,  and 
improved  quality  of  service  are  important  goals.  The  program  will  make  exten- 
sive use  of  automation  and  will  feature  direct  user  access — direct  pilot  com- 
munication with  the  system  computer  to  obtain  a weather  briefing  and/or  to  file 
a flight  plan. 

The  Voice  Response  System  envolved  into  its  present  form  after  a period  of 
controlled  experimentation  by  FAA  personnel  who  sought  to  develop  suitable 
procedures  and  to  acquire  the  necessary  weather  products.  But  realistic 
testing,  live  experimentation,  was  necessary  to  find  the  ways  in  which  to 
maximize  system  usefulness  and  convenience  of  use. 

TYPICAL  SYSTEM  TRANSACTION . 

The  VRS  enables  pilots  to  access  a computer  data  base  of  weather  and  flight 
data.  The  pilot  enters  requests  for  specific  weather  reports  for  locations 
of  interest  by  depressing  the  keys  on  a touchtone  telephone  or  touchtone  adap- 
ter on  dial-type  telephones.  Almost  immediately,  the  system  responds  with 
a spoken  message  derived  from  stored  human  speech.  The  pilot  need  only  have 
a standard  12-key  touch-tone  telephone  and  a slight  knowledge  of  the  means  of 
communicating  with  the  Voice  Response  System.  After  connection  with  the 
system,  a typical  dialogue  might  transpire  as  follows: 

VRS:  "Hello.  Greenwich  time  is  now  . Enter  location  identifier." 

Pilot:  The  pilot  depresses  the  touchtone  keys  in  the  manner  prescribed  in  his 
brochure  and  enters  location  identifier. 

VRS:  The  voice  feature  repeats  the  location  entered  either  by  pronouncing  it 
phonetically  or  as  normally  spoken. 

VRS:  "Enter  next  location  identifier." 
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Pilot:  The  pilot  may  repeat  the  location  entry  procedure  until  he  has  inserted 
as  many  as  are  necessary,  up  to  a maximum  of  10, 

VRS : "Do  you  want  hourly  observations?  Answer  yes  or  no?" 

Pilot:  Enters  numeric  9 (y)  for  yes,  6 (n)  for  no.  (9  occurs  with  y and  6 
occurs  with  n on  the  key  pad) 

VRS:  If  pilot  responds  "yes,"  requested  hourly  observations  are  delivered. 

VRS:  "Do  you  want  terminal  forecasts?  Answer  yes  or  no?" 

Pilot:  Enters  numeric  9 for  yes,  6 for  no. 

VRS:  If  pilot  responds  "yes,"  the  requested  information  is  delivered  for  those 
entered  locations  where  terminal  forecasts  are  made. 

VRS:  "Do  you  want  forecast  winds  aloft?  Answer  yes  or  no?" 

Pilot:  Enters  numeric  9 for  yes,  6 for  no. 

VRS:  "How  many  hours  from  now?  The  maximum  is  30." 

Pilot:  Enters  numeric. 

VRS:  At  what  altitude? 

Pilot:  Enters  altitude. 

VRS:  Delivers  forecast  grid  winds  aloft  for  the  entered  altitude  and  for  4,000 
feet  above  and  below  that  altitude.  (The  system  does  not  provide  data  for  an 
altitude  less  than  2,000  feet  above  the  terrain). 


The  above  example  is  a simplified  illustration  of  how  the  pilot  and  the  VRS 
communicate.  A more  detailed  account  of  how  the  system  operates,  its  control 
functions,  and  limitations  is  contained  in  figure  1.  Figure  1 is  a copy  of 
the  publicity  brochure  that  was  sent  to  approximately  8,800  general  aviation 
pilots  in  the  Washington  D.C.  metropolitan  area. 

EQUIPMENT  DESCRIPTION. 


The  VRS  provides  computer-generated  voice  output  in  response  to  inputs  from  a 
touchtone  telephone.  Stored  within  a Digital  Equipment  Corporation  PDP  11/70 
computer  were  the  executive/operational  programs  required  to  operate  the  sys- 
tem and  the  raw  national  weather  data  base  received  from  the  Weather  Message 
Switching  Center  (WMSC)  located  in  Kansas  City,  Missouri.  The  data  base  was 
translated  and  reformated  into  VRS  weather  files.  All  data  manipulations 
regarding  currency  and  validity  of  the  national  weather  data  base  were  accom- 
plished within  this  computer  which  was  located  at  the  MITRE  Corporation 
facility,  Me  Lean,  Virginia. 
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The  digitized  voice  response  subsystem  was  located  at  the  Transportation  Systems 
Center  (TSC) , Cambridge,  Massachusetts,  and  was  connected  to  the  MITRE-based 
PDP  11/70  by  a 1,200-baud  subchannel  of  a multiplexed  9,600-baud  communication 
line.  The  digital  subsystem  is  composed  of  four  major  elements;  i.e.,  a pro- 
cessor, fixed-head  disk,  and  touchtone/voice-decoding  equipment. 

The  processor  is  a Digital  Equipment  Corporation  PDP  11/34  minicomputer  with 
64k  of  core  memory.  It  performs  the  functions  of  interpreting  the  touchtone 
inputs  from  the  user,  determining  what  weather  information  has  been  requested, 
accessing  the  proper  weather  files  in  the  PDP  11/70,  and  converting  the  output 
into  speech  by  stringing  together  prestored  words  and  phrases  which  result  in 
the  appropriate  output  message. 

The  fixed-head  disk  was  used  to  store  spoken  words  and  phrases  which  had  pre- 
viously undergone  an  analog-to-digital  conversion.  The  digital  output  was 
compressed  utilizing  a technique  known  as  adaptive  differential  pulse  code 
modulation  (ADPCM) . 

The  process  of  decompressing  the  data  stored  on  the  disk  upon  retrieval  was 
performed  by  the  speech  decoding  hardware.  This  equipment  consisted  of  20  in- 
dependent decoding  channels  which  effectively  reversed  the  process  of  ADPCM. 

The  touchtone  decoding  equipment  consisted  of  20  independent  decoding  channels 
and  performed  the  function  of  decoding  the  touchtone  inputs  and  routing  the 
resultant  signals  to  the  PDP  11/34.  The  20-channel  system  uses  10  Bell  Tele- 
phone Company  407C  modems  and  a 20-channel  multiplexer  to  multiplex  these 
channels  into  one  computer  input  channel. 

For  redundancy,  a second  PDP  11/34  was  available  at  TSC.  In  the  event  of  a 
PDP  11/70  failure,  the  weather  data  base  was  buffered  at  WMSC  for  a limited 
period  of  time. 

METHODOLOGY 


SURVEY. 

The  VRS  became  operational  April  15,  1978,  in  the  test  area  which  included  the 
District  of  Columbia,  Fairfax  County,  Arlington  County,  Alexandria  City,  Prince 
Georges  County,  and  Montgomery  County.  In  April  of  1978,  a brochure  (figure  1) 
and  a wallet-size  card  (appendix  A)  were  mailed  to  approximatley  8,800 
registered  general  aviation  pilots  residing  in  the  test  area.  The  brochure 
explained  the  reasons  for  the  experiment,  what  the  Voice  Response  System  is, 
and  how  to  use  it.  An  illustrative  copy  of  the  covering  letter  accompanying 
the  brochure  and  wallet-size  card  is  shown  in  figure  2.  A random  sample 
of  1,778  names  by  license  category  was  subsequently  drawn  from  the  respondent 
universe  of  approximately  8,800  general  aviation  pilots  residing  in  the  test 
area.  During  the  week  of  July  21,  1978,  a questionnaire  (figure  3)  was  mailed 
to  each  potential  respondent  in  the  sample  along  with  another  copy  of  the 
brochure  and  wallet-size  card.  A copy  of  the  covering  letter  to  the 
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INTRODUCTION. 

The  Federal  Aviation  Administration  is  planning 
to  implement  a national  Flight  Service  Station 
Automation  Program.  This  program  is  designed  to 
meet  the  increasing  demand  tor  services  to  the 
general  aviation  fleet  without  a large  increase  in 
operating  costs.  Service  improvements  in  terms  of 
system  capacity,  accessibility,  and  quality  are  high 
on  the  list  of  system  characteristics.  The  program 
makes  extensive  use  of  automation  and  will 
introduce  the  concept  of  direct  user  access — the 
capability  whereby  a pilot  directly  accesses  an  FSS 
computer  to  obtain  a weather  briefing  and/or  file  a 
flight  plan.  Direct  pilot  input  and  computer 
generated  voice  response  are  one  of  the  primary 
means  of  providing  the  improved  service. 

BACKGROUND: 

The  Voic»  Response  System  (VRS)  enables  pilots 
to  access  a computer  data  base  of  weather  and 
flight  data.  The  pilot  inputs  requests  for  specific 
weather  reports^for  locations  of^interest  by 
utilizing  the  keys  on  a TOUCUTONE  telephone  or 
keypad  adapter  on  dial -type  telephones.  Almost 
immediately  the  system  responds  with  a computer 
generated  voice.  To  access  this  system,  the  pilot 
need  only  have  a standard  12  key  telephone  and  a 
slight  knowledge  of  the  manner  of  communicating 
with  the  voice  response  system.  Controlled 
experiments  were  conducted  by  FAA  personnel  to 
develop  and  refine  workable  procedures  and  to 
establish  the  required  weather  products. 

However,  live  experimentation  is  necessary  to 
maximize  system  usefulness  and  convenience  of 
use. 

In  the  very  near  future,  the  FAA  will  conduct  a 
live  field  experiment,  to  ascertain  pilot  reactions 
and  to  provide  a means  for  pilot  suggestions  for 
improvements.  Initially  the  weather  products 
available  will  be  Surface  Observations,  Terminal 
Forecasts;  and  Forecast  Winds  Aloft.  Thisweatl^ 
information  will  be  current  These  experiments  aW 
scheduled  to  begin  approximately  April  I.  1978 
and  will  be  conducted  over  a period  of  4 months 
in  the  Washington  JJ.C.  area. 


The  three  products  are  intended  to  be  used  for 
early  flight  planning  purposes,  providing  the  pilot 
with  "go"  or  "no  go  to  the  airport"  information. 
The  reports  will  be  checked  agauist  predescribed 
criteria  to  assure  reasonableness  of  data  - Reports 
which  are  not  received,  cannot  be  deciphered  or 
do  not  meet  predeteimined  criteria  will  be  voiced 
as  "not  available."  Since  the  system  does  not 
contain  all  the  products  necessary  for  a complete 
briefing,  contact  your  FSS  for  any  additional 
information  lequired. 


OPERATING  PROCEDURES 

Any  public,  business,  or  home  telephone  with  a 
standard  12  key  signalling  system  will  be  used  to 
access  the  system.  The  conventional  rotary  dial 
telephone  may  be  utilized  only  for  dialing  the 
access  numbers,  but  an  acoustically  coupled  lone 
signalling  device,  in  lieu  of  a TOIJCHTON1- 
telephone,  must  be  employed  in  conjunction  with 
the  rotary  dial  telephone  to  enter  the  information 
requests.  Acoustically  coupled  tone  signalling 
devices  (with  12  or  16- key  data  entry  device) 
are  generally  available  from  electronics  supply 
houses  at  low  costs  or  can  be  leased  from  your 
local  telephone  company. 

Initially,  you  access  the  Voice  Response  System 
by  dialing  the  system  access  number  of  347  3222. 
This  will  be  a toll  free  call  when  initiated  in  the 
greater  Washington  metropolitan  area.  Otherwise, 
it  is  a standard  toll  call  into  the  347  exchange  area. 

To  communicate  with  the  computer  you  must  use 
the  keypad  in  a way  that  the  computer 
“understands." 


KEYPAD 


Locations  (weather  reporting  stations  and  airpotts) 
are  uniquely  identified  by  three  letter 
combinations  and  you  enter  these  three  letter 
identifiers  to  delineate  a single  location  or  a series 
of  locations  (e  g .a  proposed  fiighlpath)  for  which 
you  desire  to  know  the  weather 


' 


The  keypad  does  not  have  enough  ke] 
the  entry  of  an  alphabetic  character  (Id 
single  keystroke.  But  it  is  possible  t< 
unambiguous  entry  by  depressing  two 
can  enter  a particular  letler  by  depress 
on  which  t hat  letter  appears  and  anol 
indicate  which  of  the  three  letters,  ii 
3rd.  The  numeral  "I"  key  indicates  thi 
the  numeral  “2”  key  indicates  the  2a 
numeral  “3"  key  mdicates  the  3rd.  Th< 
B is  signalled  by  depressing  the  key  a 
appears  (the  number  “2"  key)  and 
numeial  “2”  key  (2nd  letter  in  the  gn 

The  letter  C is  signalled  by  depressing 
which  “C”  appears  and  the  numeral  “3 
letter  in  group  ABC).  For  example.  DCi 
as  D l.C  3,  A I . as  shown  below.  | 


r 


pd  does  not  have  enough  keys  to  allow 
of  an  alphabetic  chatacter  (lettet)  with  a 
bstrokc.  But  it  is  possible  to  nuke  an 
i»us  entry  by  depressing  two  keys  You 
a particular  letter  by  depressing  the  key 
that  letter  appears  and  another  key  to 
trhich  of  the  three  letters,  1st,  2nd,  or 
| numeral  "1”  key  indicates  the  1st  letter, 
il  "2”  key  Indicates  the  2nd,  and  the 
‘3"  key  indicates  the  3rd.  Thus  the  letter 
led  by  depressing  the  key  on  which  B 
(the  number  "2"  key)  and  then  the 
“2”  key  (2nd  letter  in  the  group.  ABO. 


C is  signalled  by  depressing  the  key  on 
' appears  and  the  numeral  “3"  key  (3rd 
goup  ABC).  For  example,  DCA  is  entered 
t-3,  A 1 , as  shown  below. 


D 


PI 


ABB 

■ b a 
a n a 
□ a □ 


■ a a 

h b a 

BBS 
□ ID 


As  shown  above,  the  letters  Q and  Z and  the  blank 
character  are  assigned  to  the  numeral  “1"  key.  Q is 
1-1,  ‘blank’  is  1-2,  and  Z is  1-3.  Each  of  the 
twenty  six  letters  of  the  alphabet  can  be  entered 
in  this  fashion  (two  keystrokes)  and  no  confusion 
will  result  The  ‘blank'  is  not  used. 

But  it  does  not  suffice  just  to  be  able  to 
communicate  a string  of  letters  of  the  alphabet  to 
the  computer.  You  must  be  able  to  tell  the 
computer  what  you  want  done  with  the 
information  you  have  provided.  At  the  lower 
right-hand  corner  of  the  keypad,  there  is  a key 
imprinted  with  a symbol.  We  call  this  the 
‘computer  entry'  key  or.  for  conciseness,  the 
■pound'  key.  Since  this  key  is  not  used  to  transmit 
letters  or  numbers,  it  creates  no  confusion  to 
employ  it  as  a control  key  to  signal  an  action  or  a 
request.  Used  in  conjunction  with  other  keys,  a 
number  of  different  actions  can  be  signalled.  Other 
control  functions  will  be  explained  later. 

Some  location  identifiers  use  both  letter  and 
numerals  For  these  entries,  it  is  necessary  to 
utilize  two  keystrokes  for  each  letter  or  numeral. 
The  context  of  the  pilot  computer  dialogue  will 
often  preclude  ambiguities  and  permit  simpler  data 
entry.  Numbers  can  be  entered  unambiguously  by 
depressing  the  ‘OPFR’  key  and  the  appropriate 
numeral  key  The  *OPF.R‘  key  is  the  key 
representing  the  numeral  ‘0’  (or  zero)  so  that  entry 
of  the  numeral  f)‘  involves  two  actuations  of  the 
■OPFR-  key  The  numeral  'S'  is  communicated  by 
depressing  'OPFR'  and  'S'  (as  shown  below)  and 
the  other  numerals  are  similarly  communicated. 
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B B B 
D a □ 


B 

B B 

B 

« a 

B 

b a 

□ 

a □ 

The  procedure  described  is  used  only  for  entering 
numbers  in  three-letter  location  identifiers  with 
mixed  letters  and  numbers.  For  all  other  numeric 
entries,  single  keystrokes  for  numbers  are  required . 
For  example,  if  the  computer  ‘voice’  requests  an 
altitude  or  a number  of  hours  (from  the  present 
time),  then  the  numeric  entries  for  these  fields 
may  be  made  via  a single  keystroke  for  each  digit 
of  the  entry. 

You  will  seldom,  if  ever,  be  confused  in  practice 
and  you  will  find  that  most  of  the  numerical 
entries  you  need  to  make  will  require  only  a single 
key  actuation  per  digit.  In  the  uncommon  case 
where  you  wish  to  enter  an  identifier  such  as  6B2 
(Greenville,  Maine),  you  will  probably  experience 
no  difficulty  in  recognizing  that  the  keystroke 
sequence  should  be  0-6.  B-2,0-2. 


a a a 

b a a 

B B B . 

> □ B □ 

D 

BIB 

a a a 
b b a 

□ HQ 

2 

a a a 

a b a 
b b a 

B i D 

b a a 
a a a 
b b a 
□ a □ 

i 

FIGURE  I.  VRS  BROCHURE 
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AN  EXAMPLE  OF  A TYPICAL  VRS 
DIALOGUE: 


DATA  NOT  AVAILABLE 


The  computer  must  be  able  to  recognize  the  end 
of  an  entry  (i.e..  a string  of  alphabetic,  numeric  or 
mixed  characters)  and  the  request  that  it  respond. 
The  computer  entry  key  (V/or  'pound'  key)  is 
depressed  twice  to  provide  the  end  of-entry 
signal  immediately  following  each  and  every  field. 
Thus,  to  request  weather  data  for  Martmsburg, 
W.  Va.  (and  vicinity),  you  generate  the  keystroke 
sequence  ‘M-l\  'R  -2','B  2',  V 

The  computer  will  ‘read  back’  each  item  entered 
so  that  you  may  verify  the  correctness  of  your 
entry.  The  phonetic  alphabet  will  generally  be 
used  so  that  the  identifier  MIV  will  be  read  back  as 
"MIKE"  "INDIA”  "VICTOR"-,  CHO  will  be  read 
back  as  "CHARLIE"  "HOTEL"  “OSCAR".  For 
some  locations,  the  actual  name  of  the  airport  will 
be  read  back  to  determine  the  mode  of  response 
preferred  by  pilots.  For  example,  DCA 
(Washington  National  Airport)  will  be  read  back  as 
“Washington  National.” 

CONTROL  FUNCTIONS: 

The  use  of  the  (pound)  key  was  discussed 
previously.  Hie  (STAR)  key  is  used  to  sic*.  me 
computer  response.  While  in  the  response  mode,  if 
you  wish  to  interrupt  the  computer  voice 
response,  simply  depress  the  ‘<r  key.  You  can  halt 
the  voice  response  until  you  are  ready  to  proceed. 
After  stopping  the  response,  you  may  then  order  a 
resumption  of  voice  response,  a repeat,  a jump 
ahead  (skip),  or  a begin  over,  by  selecting  the 
appropriate  keystroke  sequence  shown  below. 
Notice  that  the  enter  command  V— V is  not 
required  after  the  control  functions  containing  the 
'Ct  (STAR)  keystroke. 


enter _0  0 RfPtAT 0 0 

YE s m _.  0 3 0 JUMP  AHCAO  _0  B 

NO  IN) E 0 0 DELETE 0 0 

STOP 0 BEGIN  OVEN 0 3 

GO 0 0 

Notice  that  “YES”  or  “NO"  only  requires  three 
keystrokes  “Y"  or  “N” 


Now  that  we  have  explained  the  procedures 
necessary  io  obtain  access  to  the  weather  briefing, 
let's  follow  a pilot  through  a typical  briefing. 

Initially,  you  access  the  Voice  Response  System 
by  dialirg  or  keying  the  following  telephone- 
number — 347  3222. 

PILOT  - pilot  dials  347-3222 
SYSTEM  “HELLO",  “Current  Greenwich  Time 
is  XXXX.” 

SYSTEM  - “Enter  Location  Identifier." 

PILOT-  (Desired  location  PIT)  P-1;  1-3; 
T—  I ; ft# 

SYSTEM  - “PAPA",  “INDIA”,  ‘TANGO” 
"ENTER  NEXT  LOCATION" 

PILOT  (Desired  kication  - ILG)  1-3;  L-3; 
G-l;  *• 

SYSTEM  - “INDIA",  “LIMA",  "GOLF' 
"ENTER  NEXT  LOCATION” 

PILOT  - (If  no  additional  entries,  enter  mt ) 
SYSTEM  - “Do  you  want  hourly  surface 
observations?  Answer  yes  or  no." 
PILOT  - Y;  » # 

SYSTEM  - reads  hourlys  for  PIT,  ILG,  etc. 
SYSTEM  - “Do  you  want  terminal  forecasts7 
Answer  yes  or  no" 

PILOT-  Y;»# 

SYSTEM  - reads  forecasts  for  PIT  and  ILG 
SYSTEM  - “Do  you  want  forecast  winds  aloft? 

Answer  yes  or  no." 

PILOT  - Y;  « # 

SYSTEM  “How  many  hours  from  now7  The 
maximum  is  30.” 

PILOT-  6;  tut 
SYSTEM  - "six" 

SYSTEM  - “At  what  altitude?" 

PILOT-  85;  (or  8500;  , no  matter)  ft* 
SYSTEM  - “eight  five” 

SYSTEM  - reads  winds  aloft  at  requested 
altitude,  7 4000  feet  and -4000  feet 
for  each  location. 

SYSTEM  - “Do  you  want  more  information? 

Answer  yes  or  no.” 

PILOT  - Y;  * a 

SYSTEM  - “Enter  location  identifier,  etc.” 


When  data  are  not  available,  one  C 
will  occur 

1 Wrong  Identifier  If  a thrce-i 
wtuch  does  not  constitute  a 
identifier  is  made  (e.g  . ABC),  th* 
back  the  characters  as  entered.  Ho* 
report  requested  is  to  be  read  out 
say  “ALPHA-BRAVO -CHARI 
location  identifier." 

2.  No  Report  for  a Given  Location 
identifier  is  a valid  one  but  not  a r 
for  the  type  of  report  requested,  t 
"ALPHA  BRAVO  CHARLIE  ...| 
Observation  Station"  or  "...  is  | 
forecast  location." 


3.  Noncurrent  Data  If  the  locatio 
valid  one  but  the  current  data  ar 
the  VRS  will  say  (l 
“SIERRA-BRAVO  YANKEE  . 
available”  for  report  type  requested 


NOTE  I HOURLY  OBSER1I 
the  latest  available  observation 
provided  that  the  observation  is  I 
hours  old.  Special  observations 
to  last  hourly. 


2.  In  this  experinH 
stations  for  weather  observed 
continental  United  States  are  coni 
base. 


3.  Minimum  altitu 
winds  aloft  is  approximately 
terrain  level. 


4.  The  system  has 
functions  which  limit  the  amol 
individual  can  use  the  system.  1 
been  incorporated  to  preclude  an 
tying  up  the  phone  lines  for  anj 


I 


DATA  NOT  AVAILABLE 

When  data  are  nol  available,  one  of  the  following 
will  occur 

1.  Wrong  Identifier  If  a three  character  entry 
which  does  not  constitute  a valid  location 
identifier  is  made  (eg.,  ABC),  the  VRS  will  read 
back  the  characters  as  entered.  However,  when  the 
report  requested  is  to  be  read  out,  the  VRS  will 
say  “ALPHA  BRAVO-CHARLIE.-js  not  a 
location  identifier." 

2.  No  Report  for  a Given  Location:  If  the  location 
identifier  is  a valid  one  but  not  a reporting  station 
for  the  type  of  report  requested,  the  VRS  will  say 
“ALPHA  BRAVO-CHARLIE  ...  is  not  an  Hourly 
Observation  Station”  or  “...  is  not  a terminal 
forecast  location." 

3.  Noncurrent  Data  If  the  location  identifier  is  a 
valid  one  but  the  current  data  are  not  available, 
the  VRS  will  say  (eg.,  SBY), 
“SIERRA-BRAVO  YANKEE  ...  report  not 
available"  for  report  type  requested. 

NOTE.  I.  HOURLY  OBSERVATIONS  Only 
the  latest  available  obser'ation  will  be  given 
provided  that  the  observation  is  not  more  than  2 
hours  old.  Special  observations  will  be  appended 
to  last  hourly. 

2.  In  this  experiment  all  reporting 
stations  for  weather  observations  within  the 
continental  United  States  are  contained  in  the  data 

base. 

3.  Minimum  altitude  for  forecasted 
winds  aloft  is  approximately  2,000  feet  above 
terrain  level. 

4.  The  system  has  some  time-out 
functions  which  limit  the  amount  of  time  an 
individual  can  use  the  system.  This  feature  has 
been  incorporated  to  preclude  an  individual  from 
tying  up  the  phone  lines  for  an  extended  period. 


HOURS  OF  OPERATION 

The  VRS  will  operate  from  6 00  A M until  8 00 
P.M.  daily.  Pilots  can  access  the  system  by  calling 
347  3222.  This  number  is  a local  call  withm  the 
Washington.  DC.  exchange  area.  Please  keep  in 
mind  that  this  system  is  experimental.  There  may 
be  periods  when  system  outages  occur.  During 
these  periods,  a message  will  be  placed  on  the 
phone  lines  to  notify  pilots  that  the  system  is 
inoperative. 

This  brochure  has  been  mailed  to  nearly  10,000 
registered  pilots  in  Washington,  D.C.  and  the 
surrounding  counties  of  Virginia  and  Maryland. 
This  service  is  tailored  for  your  needs  and  we 
request  your  participation  in  its  evaluation  and 
improvement. 

If  you  did  not  receive  a personal  copy  of  this 
brochure  or  if  you  have  any  questions,  please  send 
a stamped  self-addressed  envelope  to: 

VRS 

DOT/FAA/NAFEC 

ANA-250 

Atlantic  City,  N J.  08405 

We  sincerely  encourage  your  participation  in  this 
experiment.  Thank  you  very  much  for  your 
cooperation. 


U.S.  DEPARTMENT  OF  TRANSPORTATION 
FEDERAL  AVIATION  ADMINISTRATION 
National  Aviation  Facilities  Experimental  Center 
Atlantic  City,  New  Jersey 


Flight  Service  Station 
Automation 

and 


FIGURE  1.  VRS  BROCHURE  (Continued) 
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DEPARTMENT  OF  TRANSPORTATION 

FEDERAL  AVIATION  ADMINISTRATION 


WASHINGTON,  O.C.  20590 


April  1978 


TO:  All  Computer  Generated  Voice  Users 

SUBJECT:  Voice  Response  System  (VRS)  Demonstration 

The  Federal  Aviation  Administration  (FAA)  has  developed  an  automated 
weather  data  dissemination  system  which  allows  a pilot  to  obtain  a 
limited  weather  briefing  without  the  aid  of  a flight  service  specialist. 
This  system,  which  is  called  the  "Voice  Response  System  (VRS),"  has 
been  tested,  in  a controlled  environment,  by  FAA  personnel.  The  FAA 
is  now  ready  to  test  the  system  in  a "live”  environment.  This  test  is 
scheduled  to  begin  on  or  about  April  1,  1978,  and  will  be  conducted 
over  a period  of  4 months. 

The  purpose  of  VRS  is  to  provide  pertinent  meteorological  information 
to  the  pilot,  while  still  at  home  or  office,  through  the  use  of  a 
push-button  telephone.  The  material  presented  in  the  VRS  is  current 
weather  data  as  reported  at  the  time  of  access.  The  demonstration 
system  will  provide  three  preflight  weather  products:  hourly  surface 
observations,  terminal  forecasts,  and  forecast  winds  aloft.  These 
products  are  intended  to  be  used  for  early  flight  planning  purposes. 

Since  the  system  does  not  contain  all  the  recommended  weather  and- 
f light  data  products  required  for  a complete  preflight  briefing,  its 
use  is  not  intended  to  completely  replace  a Flight  Service  Station 
specialist  briefing.  Further  development  of  the  VRS  concept  is  under- 
way to  expand  the  capabilities  of  the  system. 

The  enclosed  brochure  contains  instructions  on  how  to  use  and  operate 
the  system.  Pilots  can  access  the  system  by  calling  347-3222.  This 
number  is  a local  call  within  the  Washington,  D.C. , exchange  area. 

VRS  operation  will  be  available  through  July  1978,  from  6:00  a.m.  until 
8:00  p.m.  daily.  Since  the  system  is  experimental  and  not  equipped  for 
extended  or  redundant  operation,  some  system  outages  will  occur. 

During  these  periods,  a message  will  be  placed  on  the  phone  lines  to 
notify  pilots  the  VRS  is  not  operating.  It  is  important  to  note  that 
use  of  the  VRS  requires  a push-button  telephone  Rotary  dial  phones 
can  only  be  used  in  conjunction  with  an  auxiliary  push-button  pad. 

These  adaptive  pads  can  be  leased  from  your  local  telephone  company  or 
purchased  from  numerous  telephone  equipment  or  electronic  supply  outlets. 


(over) 

FIGURE  2.  COVER  LETTER  TO  VRS  BROCHURE 
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Pilot  participation  in  this  demonstration  is  encouraged.  The  FAA  is 
currently  finalizing  an  activity  to  obtain  pilot  reaction  and  comments 
regarding  operational  use  of  the  VRs  during  the  demonstration.  Further 
information  concerning  this  aspect  of  the  demonstration  will  be 
disseminated  during  May  1978. 

Once  again,  we  urge  you  to  help  us  evaluate  this  new  concept.  Your 
assistance  is  welcomed  and  appreciated  by  FAA. 


Sincerely, 


Enclosure 


. ' 


FIGURE  2.  COVER  LETTER  TO  VRS  BROCHURE  (CONTINUED) 


7 


Form  Approved  OMB  No  04  S78010 


DEPARTMENT  OF  TRANSPORTATION 

f EDERAL  AVIATION  ADMINISTRATION 

VOICE  RESPONSE  SYSTEM  <VRS)  SURVEY 


This  report  is  authorized  by  the  Federal  Aviation  Act  Section  303  and  3T1  While  you  are  not  required  to  respond  your 
cooperation  is  needed  to  nsake  the  results  of  this  survey  comprehensive,  accurate,  and  timely 


What  license  rating  do  you  hold  ’ 


n StuOem 

J □ 

J □ Pi'vatr 


4 Q O" 

b Q Ci  - 
b 3 A" 


Have  you  used  the  VRS’ 

mQVes»  n ,es  estimated  No  of  times  _____ 
i2'Q\  V-  ii'uthii'i'e  pi'. >m-  available 
i3’Dn.  pi.  ■ase  i' » plain  m block  \2  tie 1 1 1 a 
It  NO.  piiMsi-  r .■turn  v our  . . >oiv<  jt.cn>  a appreciated 


Total  (lying  time  Knurs 

Ni  d hours  dur  mg  the  past  1 ? months.. 


Your  city  and/or  county  of  residence  - 

Where  do  you  normally  get  your  weather  briefing7 


J [J  \..t . 

I 3 Olhr 


•"vir  Station 
WmIIii  Sr- . 
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Place  a check  mark  in  the  column  which  best  describes 
the  extent  to  which  you  agree  with  each  of  the  following 
statements  as  they  apply  to  your  general  use  of  the  VRS 
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Did  the  briefing  received  from  the  VRS  make  d 
unnecessary  for  you  to  call  the  Flight  Service 
Station  iFSSl  tor  weather  information’ 


a Fi» 

111  Q • ■ 121  J x 


m □ • <2i ; 


If  you  still  had  to  call  the  F SS  for  Aeather  information 
did  the  VRS  briefing  reduce  the  time  on  the  line  with 
the  FSS7 


<i>3 


121  □ X 


Of  the  two  readhack  methods  for  location 
identifiers  which  cfo  you  prefer  ' 


iii  □ 


!?'  □ 


Spoken  name 
W I . P n*burgh 


10  Based  on  your  e*perience  with  the  VRS  would  you 
use  it  if  implemented  permanently7 


1 D r , . 

2 □ fo.  . 

3 □ . ... 
. O X.,. 


1 1 Suitability  of  the  VRS  for  wide  spread  public  use 


1 Q Sociable  a % 

Q Suitable  '"•'i'  ' • nanqps  (leviable 
1 Q Me  j • .1  , V uitabo  main*  i hinges  necesaa'v 
4 D IJnsuiiaMe 


If  then-  are  any  comments  or  suggestions  you  would  like  to  nsake  regarding  the  VRS  please  reesrd  them  here 


F A A FORM  NA  M90  ! (Use  r*  pi  rat  Sept  T97RI 

FIGURE  3.  VRS  QUESTIONNAIRE 
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questionnaire  is  shown  in  figure  4.  Subsequent  followup  letters  to  nonrespond- 
ents are  shown  in  appendix  A.  A copy  of  the  letter  mailed  to  all  pilots  not 
selected  for  the  VRS  survey  is  shown  in  appendix  A. 

Table  1 summarizes  the  VRS  test  procedures  and  time  periods.  The  summary  of 
the  questionnaire  returns  by  individual  mailing  and  pilot  license  category  is 
shown  in  table  2. 

VRS  QUESTIONNAIRE. 

The  VRS  questionnaire  uses  the  Likert  technique  (reference  1).  The  Likert-type 
questionnaire  consists  of  statements  which  are  presented  to  the  subject  with 
instructions  that  he  is  to  indicate  his  level  of  agreement  or  disagreement 
to  each  statement.  The  subject  does  so  by  marking  one  of  five  or  seven 
places  on  a vertical  scale  ranging  from  "strongly  or  completely  agree"  to 
"strongly  or  completely  disagree."  The  agree/disagree  version  of  the  Likert- 
type  scale  has  undergone  a number  of  years  of  research  since  it  was  first  pub- 
lished by  Rensis  Likert  in  the  early  30' s.  It  has  been  used  extensively  in 
attitude  research.  The  Likert-type  scale  was  used  in  the  New  York  City  PATWAS 
questionnaire  with  good  results  (reference  2). 

The  VRS  questionnaire  contains  some  items  that  are  stated  in  the  positive  form 
and  others  in  the  negative.  It  is  in  accord  with  the  best  current  design  prac- 
tice to  vary  the  dimensionality  of  the  "favorable"  or  approve  response  category. 
As  stated  by  experimental  psychologist,  Dr.  Richard  Sulzer  of  NAFEC: 

"A  well  constructed  questionnaire  or  self-report  scale  should  contain  safe- 
guards against  common  sources  of  error,  sucli  as  the  "halo  effect"  (the  tendency 
to  be  unduly  influenced  by  a single  or  overall  favorable  aspect  when  rating 
separate  dimensions). 

"If  the  system  being  evaluated  is  viewed  positively,  and  the  first  several 
i“'m<s  call  for  a "highly  approve"  rating,  there  is  a tendency  for  later  items 
to  be  rated  in  the  same  way  "highly  approve"  or  "strongly  agree  that  it  is 
good,"  despite  the  fact  that  on  one  or  more  of  these  later  items,  there  might 
be  a tendency  toward  less  actual  agreement  or  approval. 

"In  the  VRS  questionnaire,  we  are  seeking  independent  ratings  on  various 
factors.  It  is  quite  reasonable  to  assume  that  some  aspects  of  the  system  are 
actually  better  than  others.  Some  warrant,  no  doubt,  universal  approval; 
others  may  be  capable  of  improvement.  We  want  the  respondents  to  read  and 
respond  to  each  item  independent  of  the  previous  item  or  of  any  overall  (halo) 
attitude  that  applies  to  the  VRS  system." 

SAMPLE  SIZE. 

Using  computer-generated  random  numbers,  a random  sample  of  1,778  names  by 
license  category  was  drawn  from  the  respondent  universe  of  approximately 
8,800  pilots.  A smaller  permissible  error  of  estimation  was  established  for 
private  and  commercial  pilots,  since  they  represent  the  critical  user  group. 
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DEPARTMENT  OF  TRANSPORTATION 

FEDERAL  AVIATION  ADMINISTRATION 


WASHINGTON.  D.C.  20990 


Form  Approved 
OMB  No.  04-S78010 
Use  Expires  September 


July  1978 


TO:  All  Computer  Generated  Voice  Users 

SUBJECT:  Voice  Response  System  (VRS)  Survey 

The  Federal  Aviation  Administration  is  engaged  in  an  operational 
demonstration  of  a Computer  Generated  Voice  Response  System.  You 
have  been  selected  from  a random  sample  to  participate  in  a pilot 
survey  regarding  the  feasibility  of  the  concept. 

The  VRS  is  one  of  the  few  projects  where  you  may  actually  "see”  the 
benefits  derived  from  automation.  Since  you  will  access  the  system 
directly,  your  needs  must  be  satisfied  and  the  system  should  be 
configured  (as  much  as  possible)  tc  be  acceptable  to  you.  We  need 
the  benefit  of  your  experience  through  informed  criticism  and  comments 
to  improve  the  system  design  and  performance.  This  system  will  serve 
a wide  range  of  users  with  different  experience  levels,  equipment, 
requirements,  etc.,  so  we  must  determine  the  format  and  information 
content  that  best  serve  all  users  in  terms  of  safety,  efficiency  and 
economy . 

As  a participant  in  this  demonstration,  we  are  asking  you  to 
voluntarily  complete  the  enclosed  FAA  Form  NA-6490-1,  Voice  Response 
System  Survey.  This  form  is  a postage-paid,  self-addressed  mailer. 
Please  leave  your  address  label  on  the  survey  such  that  we  can 
accurately  determine  the  sample  response.  After  you  have  completed  all 
the  items,  fold  at  the  fold  marks,  tape  it  closed  and  drop  it  in  any 
United  States  mailbox.  Since  we  do  have  a time  limit  on  this  survey, 
we  would  appreciate  its  return  within  fourteen  (14)  days. 

Your  assistance  is  welcomed  and  appreciated  by  the  Federal  Aviation 
Administration. 

Sincerely , 

CPavw&J  SK-tilL 

DAVID  J.  SHE9TEL 
• Director,  Systems  Research 
and  Development  Service 

Enclosure 


FIGURE  4.  COVER  LETTER  TO  VRS  QUESTIONNAIRE 
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Prior  to 

April  1978 

Planning  and  Design 

March  15,  1978 

- 

Distribution  of  Announcements 
of  Test  and  Descriptive  Material 

April  1,  1978 

- 

Equipment  Operational 

July  21,  1978 

July  23,  1978 

Initial  Distribution  of 
Questionnaire 

August  11,  1978 

August  12,  1978 

First  Followup  Mailing  of 
Questionnaire 

August  31,  1978 

September  1,  1978 

Second  Followup  Mailing  of 
Questionnaire 

September  18 
1978 


September  29,  1978 


Telephone  Contacts  with  Random 
Sample  of  Nonrespondents 


TABLE  2.  SUMMARY  OF  QUESTIONNAIRE  RETURNS 


For  airline  transport  rating  (ATR)  pilots,  a less  stringent  permissible  error 
was  justified  on  the  basis  that  these  pilots,  for  the  most  part,  use  other 
sources  to  obtain  weather  information  and  thus  do  not  represent  a critical  user 
group.  Similarly,  a 10-percent  permissible  error  was  allowed  for  student  pilots 
on  the  basis  that  the  student  pilot  is  not  yet  fully  capable  of  making  autonomous 
decisions  regarding  the  weather.  The  sample  size  was  determined  by  the 
formula 


as  follows: 


1 + Ne2 


Permissible 


Sample 


Since  it  was  estimated  that  more  than  50  percent  of  the  potential  respondents 
did  not  have  immediate  access  to  a touchtone  telephone,  it  was  necessary  to 
double  the  calculated  sample  size  to  a total  of  1,778.  Final  sample  size  of 
license  pilot  categories  was  established  as  follows: 


License 

Student 

Private 

Commercial 

ATR 

Total 


Sample 


1,778 


The  sample  was  not  constrained  by  county  of  residence,  since  the  process  of 
randomization  would  ensure  a representative  sample  by  county. 

DERIVATION  OF  FORMULA  FOR  DETERMINING  SAMPLE  SIZE. 


appears  in  "Statistics"  by  Taro  Yamane,  2nd  edition,  as  follows: 

For  a large  n where  the  normal  approximation  is  applicable,  the  95-percent 
confidence  interval  is 


p — 1 .96cr p <7 r <.  p -\-  1.96o-„ 


where  crp  is 


_ hr(l-Tr) 
V n 


Equation  (1)  may  be  rewritten  as 

— 1 .90 < p — v < 1 ,90o-p 

Using  the  notation  e = p - tt  , equation  (2)  can  be  rewritten  as 

— 1 .96o-p  < e < 1 .96cr, 

Considering  the  upper  tail,  we  can  set 

e — 1 .96crp 

By  squaring  both  sides  we  obtain 


tt(  t — it) 
e-  — (1.9fi)2  — — - — i 


If  we  include  the  finite  population  correction  we  get 


c*  = ( 1 99)- 


77(1  -77)  N — n 


Solving  this  equation  for  n we  obtain 


(1,99)V(1  - ir)N 
(1  9fi)-7r(  1 - 77)  + \'e- 


Since  n is  usually  unknown,  formula  (5)  is  changed  as  follows 


(1.96)*N 


(1.96)*  + 


77(  1 — 77) 


If  we  set  = 1/2,  we  know  that  the  sample  size  will  in  general  be  larger  than 
necessary.  Substituting ^ = 1/2  into  (5)  we  get 


(l.%)-(0.5)-W 

(1.96)*(0.5)*  +~\7- 


(7) 


As  further  simplification,  Yamane  uses  2 normal  deviates  instead  of  1.96  normal 
deviates.  Yamane  points  out  that  when  2 is  used  instead  of  1.96,  a larger 
sample  will  be  obtained,  and  the  confidence  coefficient  becomes  larger.  Then 
2 normal  deviates  will  provide  a 95 . 44-percent  confidence  interval  instead  of 
a 95-percent  confidence  interval.  It  should  be  noted,  however,  that  this  slight 
difference  will  have  no  significant  effect  in  practical  applications.  Thus, 
if  we  substitute  2 into  equation  (7)  we  get 


(2)*(0.5)-A’ 
(2)*(0.5)*  + Ne 1 


1 + S'e- 

A tabulation  of  this  formula  was  compiled  by  the  Bureau  of  Research,  New  York 
State  Division  of  Housing  and  Community  Renewal,  "A  Method  for  Employing 
Sampling  Techniques  in  Housing  Surveys." 


RESULTS  AND  INTERPRETATION 


RESPONSES  TO  VRS  QUESTIONNAIRE. 

PERCENTAGE  DISTRIBUTION  OF  RESPONSES  TO  QUESTION  6.  Table  3 summarizes  the 
percentage  distribution  of  responses  to  each  of  the  10  items  comp  Lsing 
question  6.  The  circled  percentages  represent  the  dominant  responses  to  each 
of  the  10  stimulus  items.  The  number  of  respondents  producing  each  of  the 
percentages  is  shown  in  table  4.  The  percentage  distribution  of  responses  to 
question  6 are  depicted  graphically  in  figure  5. 

PERCENT  OF  RESPONSES  FAVORABLE  TO  VRS.  As  indicated  in  table  3,  all  of  the 
dominant  responses  to  question  6 are  favorable  to  the  VRS.  As  shown  in 
figure  6,  82.72  percent  of  all  responses  to  question  6 are  favorable  to  the 
VRS. 

This  percentage  is  obtained  by  adding  together  the  number  of  responses  in  the 
Agree  and  Strongly  Agree  columns  for  items  6a,  6b,  6d , 6g,  and  6j  plus  the 
number  of  responses  in  the  Disagree  and  Strongly  Disagree  columns  for  items 
6c,  6e,  6f , 6h,  and  6i  and  dividing  the  result  by  the  total  number  of  responses. 
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TABLE  3.  RESPONSES  TO  QUESTION  6 — PERCENTAGES 


TABLE  A.  RESPONSES  TO  QUESTION  6 — NUMBER  OF  RESPONDENTS 


r 


REACTIONS  U • RS  STATEMENTS 


THE  ORDER  IN  WHICH  THE  INFORMATION 
WAS  PRESENTED  WAS  SATISFACTORY 


THE  INFORMATION  W AS  SUFFICIENT  TO 
MAKE  A DECISION  TO  OO  TO  THE  AIRPORT 


THE  SPEECH  RATE  WAS  TOO  FAST 


THE  INFORMATION  PROVIDES  A CLEAR 
MENTAL  PICTURE  OF  THE  WEATHER 


NOT  ENOUGH  INFORMATION  WAS  PROVIDED 


THE  V RS  OPERATING  PROCEDURES  A 
DIFFICULT 


THE  SPEECH  WAS  INT  E LLI.B  Lt 


THE  SPEECH  RHYTHM  AND  INTONATION 
WAS  UNNATURAL 


THE  SPEECH  ON  THE  VRS  WAS  UNPLEASAN 


THE  BRIEFING  WAS  SATISFACTORY 


I P) 

( tfuT 


7b 


3 S5.  S3 


I [SSSSSJk'. 

\ kj  :.SS 


3 51.00 


] 53.  50 


I *•  & » 
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V « a.»V  •'.•'t  'f' «* 3 

: : : ; 3 »». :» 


l ! I NO 


| STRONGLY 
I DISAGREE 


□ DISAGREE 
[ ^UNCERTAIN 


ag  ref 
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( U 0.  bb 

( kr 

' PM-  4 

\ L 1 ■ yj*  
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—\ 
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(PERCENT)  79-14-1 

FIGURE  5.  RESPONSE.  • V KS  STATEMENTS  IN  QUESTION  6 
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FIGURE  6.  RESPONSES  TO  QUESTION  6,  FAVORABLE  VERSUS  UNFAVORABLE 


While  all  of  the  stimulus  items  in  question  6 produced  a favorable  response 
to  the  VRS,  the  range  of  individual  responses  varied  considerably.  item  6i, 
"The  Speech  on  the  VRS  was  unpleasant"  produced  the  highest  favorable  response 
(Disagree  60.44%  + Strongly  Disagree  15.82%  = 96.26%).  Item  6e,  "Not  enough 
information  was  provided,"  produced  the  lowest  favorable  response  (Disagree 
51.00%  + Strongly  Disagree  8.20%  = 59.20%).  Figure  7 shows  the  percentage 
difference  in  favorability  between  each  stimulus  item  in  question  6.  As  shown 
in  figure  6,  82.72  percent  of  all  responses  to  question  6 were  favorable  to 
the  VRS,  9.71  percent  were  unfavorable,  and  7.57  percent  were  uncertain.  It 
should  be.  noted  that  while  item  6g,  "The  Speech  Was  Intelligible,"  yielded  a 
78.68-percent  favorable  response,  it  produced  the  highest  intense  reaction — 
"Strongly  Agree"  = 36.26  percent. 

POPULATION  CONSISTENCY.  Tables  5 through  14  and  figures  8 through  17  compare 
the  responses  to  question  6 from  three  populations,  which  consist  of  respon- 
dents to  the  (1)  initial  mailing,  (2)  first  followup,  and  (3)  second  followup 
group.  The  three  separate  populations,  which  covered  a comparable  sample  of 
individuals  in  three  separate  mailings  gave  approximately  the  same  results. 

A chi-square  analysis  showed  that  there  is  no  statistically  significant 
difference  between  the  distribution  of  responses  to  question  6 from  the  three 
populations.  All  chi-square  tests  were  conducted  at  the  .05  level  of  signif- 
icance. In  all  cases  in  which  the  three  populations  were  compared,  the  null 
hypothesis  (i.e.,  no  significant  difference)  was  accepted.  A Kolmogorov- 
Smirnov  test,  using  the  formula 


confirmed  this  result.  It  should  be  noted  that  the  greater  the  correspondence 
between  the  results  from  the  different  samples,  the  more  confidence  that  can 
be  placed  in  the  findings.  In  addition,  the  finding  that  no  statistically 
significant  difference  exists  between  the  three  mailings  not  only  increases 
the  reliability  of  the  data  but  also  shows  that  the  data  obtained  from  the  three 
mailings  can  be  combined . 

SMALL  EXPECTED  VALUES. 

The  chi-square  (>^)  distribution  is  made  under  the  assumption  that  the  expected 
values  are  not  too  small.  What  constitutes  an  expected  value  that  is  too  small 
is  somewhat  controversial  and  imprecise.  It  has  been  commonly  stated  in  the 
literature  that  expected  values  should  be  greater  than  5 for  the  chi-square 
test  to  be  valid.  Some  writers,  however,  have  expressed  the  opinion  that  this 
criterion  is  too  stringent,  and  have  suggested  less  conservative  criteria. 

A common  procedure  for  overcoming  this  problem  has  been  to  combine  categories 
with  small  expected  values.  However,  as  F.veritt  (reference  3)  has  pointed  out, 
"Such  a procedure  may  be  criticized  for  several  reasons.  Firstly,  a consider- 
able amount  of  information  may  be  lost  by  the  combination  of  categories,  and 


' 


TABLE  9.  RESPONSES  TO  QUESTION  6e 


I >t.il  1)1  26.92  24'  54.30  29  6.40  12  7.06  15  3.31  453  100.00 


TABLE  11.  RESPONSES  TO  QUESTION  6g 


Total  6 1.12  23  5.05  35  7. *9  288  63.30  103  22.64  455  100.00 


THE  ORDER  IN  WHICH  THE  INFORMATION  WAS  PRESENTED  WAS  SATISFACTORY 


DIS4GREE  AGREE 


FIGURE  8.  RESPONSES  TO  QUESTION  6a 

THE  INFORMATION  WAS  SUFFICIENT  TO  MAKE  A DECISION  TO  GO  TO  THE  AIRPORT 
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SECOND  FOLLOWUP 


FIGURE  9.  RESPONSES  TO  QUESTION  6b 
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THE  SPEECH  RATE  WAS  TOO  FAST 


INITIAL  MAILING 
FIRST  FOLLOWUP 
SECOND  FOLLOWUP 


7.-1,-. 


FIGURE  10.  RESPONSES  TO  QUESTION  6c 


THE  INFORMATION  PROVIDES  A CLEAR  MENTAL  PICTURE  OF  THE  WEATHER 
80 
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FIGURE  II.  RESPONSES  TO  QUESTION  6d 
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NOT  ENOUGH  INFORMATION  WAS  PROVIDED 
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FIGURE  12.  RESPONSES  TO  QUESTION  6e 


THE  vrs  operating  procedures  were  difficult 
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FIGURE  13.  RESPONSES  TO  QUESTION  6f 


THE  SPEECH  WAS  INTELLIGIBLE 
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FIGURE  14.  RESPONSES  TO  QUESTION  6g 

THE  SPEECH  RHYTHM  AND  INTONATION  WAS  UNNATURAL 
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FIGURE  16.  RESPONSES  TO  QUESTION  6i 


80 


THE  BRIEFING  WAS  SATISFACTORY 


me 


this  may  detract  greatly  from  the  interest  and  usefulness  of  the  study. 

Secondly,  the  randomness  of  the  sample  may  be  affected.  The  whole  rationale 
for  the  chi-square  test  rests  on  the  randomness  of  the  sample  and  that  the 
categories  into  which  the  observations  may  fall  are  chosen  in  advance.  Pooling 
categories  after  the  data  are  seen  may  affect  the  random  nature  of  the  sample, 
with  unknown  consequences.  Lastly,  the  manner  in  which  categories  are  pooled 
can  have  an  important  effect  on  the  inferences  drawn.  The  practice  of  combining 
classification  categories  should  therefore  be  avoided  if  at  all  possible."  In 
light  of  the  above,  small  expected  values  were  not  combined.  Instead,  those 
classification  categories  that  contained  none  or  only  a few  responses  were 
eliminated.  A chi-square  test  was  then  made  based  on  the  categories  containing 
the  preponderance  of  responses. 

NECESSITY  FOR  CONTACTING  FSS  AFTER  VRS  BRIEFING. 


Tables  15  and  16  and  figures  18  and  19  compare  the  responses  obtained  from 
the  three  mailings  to  question  7,  "Did  the  briefing  received  from  the  VRS 
make  it  unnecessary  for  you  to  call  the  Flight  Service  Station  (FSS)  for 
weather  information?"  The  question  elicited  a response  for  both  a preflight 
and  in-flight  status.  The  three  mailings,  which  covered  a comparable  sample 
of  individuals,  gave  approximately  the  same  results.  A chi-square  analysis 
shows  that  there  is  no  statistically  significant  difference  between  the 
distribution  of  responses  from  the  three  populations. 

Preflight 

5.21 

In-Flight  5.17 

At  the  .05  level  of  significance,  with  four  degrees  of  freedom,  the  null 
hypothesis  is  accepted;  i,e.,  no  significant  difference. 

Table  17  and  figure  20  show  that  for  preflight  planning,  56.28  percent  of  the 
respondents  answered  "yes"  to  question  7,  "Did  the  briefing  received  from 
the  VRS  make  it  unnecessary  for  you  to  call  the  Flight  Service  Station  (FSS) 
for  weather  information?"  Since  the  VRS  briefing,  which  contained  only  three 
weather  products,  was  not  intended  to  be  a complete  weather  briefing,  the 
large  number  ot  affirmative  answers  to  this  question  was  unexpected.  In 
an  effort  to  examine  this  result  in  more  depth,  the  responses  to  question 
7 were  categorized  according  to  the  following  subgroups: 

1.  License 

2.  Rating 

3.  Total  Flying  Time 

Table  18  and  figure  21  show  the  distribution  of  responses  to  the  preflight 
segment  of  question  7 by  type  of  pilot  license.  A chi-square  test  showed 
that  there  is  no  statistically  significant  difference  by  license  category. 
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TABLE  15.  RESPONSES  TO  QUESTION  7,  PREFLIGHT 


FSS  UNNECESSARY  AFTER  VRS  BRIEFING 


1 PREFLIGHT 


INITIAL  MAILING 
FIRST  FOLLOWUP 
SECOND  FOLLOWUP 


FIGURE  18.  RESPONSES  TO  QUESTION  7,  P RE FLIGHT 


FSS  UNNECESSARY  AFTER  VRS  BRIEFING 


IN-FLIGHT 
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NO  I UNCERTAIN 


INITIAL  mailing 
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SECOND  FOLLOWUP 


FIGURE  19.  RESPONSES  TO  QUESTION  7,  IN-FLIGHT 


■ , 


Table  l1*  and  figure  22  compare  the  distribution  of  responses  by  weather  rating 
In  this  case,  chi-square  is  equal  to  4.48,  which  is  not  significant  at  the 
.05  level.  Hie  null  hypothesis  is  accepted. 

Table  20  and  figure  ’ 3 show  the  distribution  of  responses  by  total  flying  time. 
As  in  the  case  of  license  category  and  weather  rating,  the  result  was  not 
statistically  significant.  Chi-square  is  equal  to  3.25  with  8 degrees  of 
freedom.  Thus,  the  unexpectedly  large  number  of  affirmative  answers  to  ques- 
tion 7 (preflight)  was  not  unduly  influenced  by  a specific  license  category, 
weather  rating,  or  by  a given  experience  level. 

Tables  21  and  22  compare  the  responses  to  question  8 obtained  from  respondents 
to  the  (1)  initial  mailing,  (2)  first  followup,  and  (3)  second  followup.  The 
three  separate  mailings,  which  covered  a comparable  sample  of  individuals, 
gave  approximately  the  same  results.  A chi-square  test  showed  that  there  is 
no  statistically  significant  difference  between  the  d Istrihution  of  responses 
from  the  three  populations. 


Pref 1 ight 

9 

= 0.27 

In- FI ight 

= 0.40 

Table  23  indicates  that  over  90  percent  of  the  respondents  answered  "yes"  to 
question  8 (preflight),  "It  you  still  had  to  call  the  FSS  for  weather  infor- 
mation, did  the  VRS  briefing  reduce  the  time  on  the  line  with  the  FSS?"  (See 
figure  24).  It  was  estimated  that  the  VRS  briefing  reduced  the  time  on  line 
with  the  FSS  by  an  average  of  47.19  percent.  The  reduction  for  in-flight  was 
not  as  striking  but  nevertheless  was  quite  substantial.  See  figure  25  for  a 
comparison. 

READ BACK  METHODS  FOR  LOCATION  IDENTIFIERS. 


Table  24  shows  the  distribution  of  responses  from  the  three  mailings  to 
question  9,  "(Of  the  two  readback  methods  for  location  identifiers,  which  do 
you  prefer?)."  As  in  all  previous  cases,  there  is  no  statistically  significant 
difference  between  the  three  populations  in  response  to  question  9.  The  value 
of  chi-square  in  this  case  is  2.67:  therefore,  the  null  hypothesis  is  accepted. 


As  shown  in  figure  26,  a large  majority  ot  respondents  in  each  of  the  three 
populations  preferred  the  spoken  name  over  phonetics.  Figure  27  shows  that 
a total  of  80.34  percent  of  .ill  respondents  preferred  the  spoken  name  as 
compared  to  19.66  percent  that  preferred  the  use  of  phonetics. 

PERMANENT  USE  OF  VRS. 


Figure  28  depicts  the  overall  response  to  question  10,  "Based  on 
ence  with  the  VRS,  would  you  use  it  it  implemented  permanently?" 
60  percent  indicated  that  they  would  use  the  VRS  tor  an  overview 
conditions.  Twenty-six  percent  indicated  that  they  would  use  it 
weather  briefing.  Since  the  VRS  w.e.  not  designed  to  provide  the 


your  experi- 
Approximately 
of  weather 
for  a complete 
pilot  with 


TABLE  19.  RESPONSES  TO  QUESTION  7,  BY  WEATHER  RATING,  PREFLIGHT 


TABLE  20.  RESPONSES  TO  QUESTION  7,  BY  PILOT  TOTAL  FLYING  HOURS,  PREILIGHI 


FIGURE  23.  RESPONSES  TO  QJESTION  7 BY  PILOT  FLIGHT  EXPERIENCE,  PREFLIGHT 


TABLE  22.  RESPONSES  TO  QUESTION  8,  IN-FLIGHT 


DID  VRS  BRIEFING  KiDUCI.  TIME  ON  LINE  WITH  FSS ? 


PRE-FLIGHT 


IN-FLIGHT 


TOTAL 


100  75  50  25  0 25  50  75 


PERCENT 


79-14-20 


FIGURE  24.  RESPONSES  TO  QUESTION  8 (PREFLIGHT/INFLIGHT) 


AVERAGE  OF  PERCENT  TIME  REDUCED 
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in-flight  40. 19 
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FIGURE  25.  RESPONSES  TO  QUESTION  8 (PERCENT  TIME  REDUCED) 


5 


READBACK  METHODS  FOR  LOCATION  IDENTIFIERS 
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FIGURE  27.  RESPONSES  TO  QUESTION  9,  COMBINED  PREFERENCE 
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FIGURE  2b.  OVERALL  RESPONSES  TO  QUESTION  10 


a complete  weather  briefing,  this  result  was  unexpected.  In  an  effort  to 
examine  the  26-percent  figure  in  more  detail,  the  responses  to  question  10 
were  categorized  according  to  the  following  subgroups: 


1.  License, 

2.  Rating, 

3.  Total  Flying  Time,  and 

4.  Number  of  Times  VRS  Used 

Table  25  and  figure  29  show  the  distribution  of  responses  to  question  10  by 
type  of  pilot  license.  A chi-square  test  showed  that  there  was  no  statistically 
significant  difference  by  license  category.  The  same  result  was  achieved  for 
weather  rating,  flying  time,  and  levels  of  VRS  usage.  Table  26  and  figure  30 
compare  VFR  and  IFR.  A comparison  of  responses  by  total  flying  time  is  shown 
in  table  27  and  figure  31. 

A comparison  between  those  respondents  that  used  the  VRS  10  or  less  times  and 
those  that  used  the  system  more  than  10  times  is  shown  in  table  28  and  figure 
32.  The  findings  show  that  the  unexpected  26  percent  who  indicated  that  they 
would  use  the  VRS  for  a complete  weather  briefing  did  not  comprise  a dispro- 
portionate number  of  a given  license  category,  weather  rating,  experience 
level  or  VRS  usage  level. 

Table  29  and  figure  33  show  the  distribution  of  responses  to  question  10  from 
the  initial  mailing,  first  followup,  and  second  followup.  It  is  apparent  that 
each  population  responded  similarly.  A chi-square  test  confirmed  that  the 
three  populations  could  be  combined. 

SUITABILITY  OF  VRS  FOR  WIDESPREAD  PUBLIC  USE. 

Figure  34  shows  the  overall  response  to  question  11  of  the  VRS  questionnaire, 
"Suitability  of  the  VRS  for  wide-spread  public  use."  Approximately  54  per- 
cent of  the  respondents  indicated  that  the  VRS  was  "Suitable,  minor  changes 
desirable."  Thirty-nine  percent  indicated  that  the  VRS  was  "Suitable,  as  is." 
Thus,  a total  of  93  percent  (92.87  percent)  of  the  respondents  to  question  11 
indicated  that  the  VRS  is  either  suitable  as  is  or  suitable  with  minor  changes. 
However,  an  examination  of  the  data  shows  that  16.67  percent  of  the  ATR  pilots 
(9  out  of  54)  indicated  that  the  VRS  is  either  unsuitable  or  requires  major 
changes.  This  percentage  is  higher  than  all  other  license  categories  combined 
as  shown  as  follows: 

License  VRS  Unsuitable  or  Requires  Major  Changes 

Students  0%  (0  out  of  24) 

Private  7.04%  (14  out  of  199) 

Commercial  5.23%  (9  out  of  172) 

ATR  16.67%  (9  out  of  54) 
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Totals  144  25.99  114  40. 45  47  12.09  7 1.24  554  100.00 


TABLE  28.  RESPONSES  TO  QUESTION  IQ,  UNDER  VERSUS  OVER  10  TIMES  USED 

Permanent  Use  of  the  VKS 

For  an  Update 


PERMANENT  USE  OK  THE  VRS 


/>  3 

i 2 . 


□ COMPLETE  WX 
BRIEFING 


Ml 

□ UPDATE 

□ NOT  AT  ALL 


20  30  40 


FIGURE  32.  RESPONSES  TO  QUESTION  10,  OVER  VERSUS  UNDER  10  TIMES  USED 
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FIGURE  33.  RESPONSES  TO  QUESTION  10  BY  MAILING  SAMPLE 
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FIGURE  34.  OVERALL  RESPONSES  TO  QUESTION  11 


A further  examination  of  the  data  reveals  that  16.25  percent  of  the  pilots 
(13  out  of  80)  with  more  than  4,800  hours  of  total  flight  time  indicated  that 
the  VRS  is  either  unsuitable  or  requires  major  changes. 

Flying  Hours  VRS  Unsuitable  or  Requires  Major  Changes 


0-160 

3.37% 

(3  out  of  89) 

161-800 

7.69% 

(12  out  of  156) 

801-1800 

3.70% 

(2  out  of  54) 

1801-4800 

2.86% 

(2  out  of  70) 

4081+ 

16.25% 

(13  out  of  80) 

Table  30  and  figure  35  show  the  distribution  of  responses  to  question  11  by 
type  of  pilot  license.  Table  31  and  figure  36  compare  the  responses  to  question 
11  by  weather  rating.  A comparison  of  responses  bv  flying  time  is  shown  in 
table  32  and  figure  37.  Table  33  and  figure  38  show  a comparison  between  two 
levels  of  VRS  usage  (<_  10  vs.  >10).  Table  34  and  figure  39  show  the  distribu- 
tion of  responses  to  question  11  by  individual  mailing.  It  is  apparent  that 
each  population  responded  similarly,  which  has  consistently  been  the  case.  A 
chi-square  test  confirmed  that  the  results  of  the  three  populations  could  be 
combined . 

WRITE-IN  COMMENTS . 

Item  12  of  the  VRS  questionnaire  invited  respondents  to  write  in  any  comments 

they  wished  to  express  regarding  the  VRS.  Figure  40  provides  a frequency 

distribution  of  the  write-in  comments.  Comments  received  in  item  12  were 
placed  in  one  or  more  of  the  categories  shown  in  figure  40.  The  number  of  times 
a given  type  of  comment  occurred  is  also  shown.  Complimentary  remarks  about  the 
VRS  constituted  the  most  frequent  comment  received.  This  type  of  comment 
occurred  227  times.  A given  respondent  may  have,  in  item  12,  commended  the 
system,  suggested  the  addition  of  a synopsis,  and  noted  that  he  prefers  plain 
language  to  phonetics.  In  such  a case,  his  remarks  would  have  been  divided 
into  three  catagories  as  follows: 

Commending  the  system  1 

Add  synopsis  of  weather  1 

Prefer  plain  language  to  phonetics  1 

The  system  of  assigning  verbal  output  to  various  categories  is  the  principal 
procedure  used  in  content  analysis.  Content  analysis  is  defined  as  a "a  method 
of  studying  and  analyzing  communications  in  a systematic,  objective,  and  quan- 
titative manner  for  the  purpose  of  measuring  variables.”  (See  Kerlinger, 
reference  4). 

The  following  statements  are  a sample  of  the  types  of  comments  falling  into  six 
selected  categories. 

COMMENDING  THE  SYSTEM.  "Much  easier  to  use  the  VRS  for  periodic  checks  of 
current  conditions  rather  than  waiting  long  times  to  talk  to  FSS  briefers." 
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TABLE  30.  RESPONSES  TO  QUESTION  11,  BY  PILOT  LICENSE  TYPE 


FIGURE  36.  RESPONSES  TO  QUESTION  11,  IFR  AND  VFR 


Im  ludes  only  the  respondents  who  Indicated  tin*  number  of  times  they  used  the  VRS. 


79-14-34 

FIGURE  38.  RESPONSES  TO  QUESTION  11,  OVER  VERSUS  UNDER  10  TIMES  USED 
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"I  think  it  is  great  and  should  save  much  time  for  pilots  and  weather 
briefers  alike." 

"A  much  needed  asset  to  the  DCA  area  due  to  the  heavy  congestion  on  the  phone 
to  the  FSS." 

"I  believe  this  to  be  a very  useful  tool  to  the  pilot,  and  an  invaluable 
service  when  he  is  looking  at  many  locations  and  an  alternate  for  each." 

"I  didn't  realize  how  much  information  I was  missing  in  my  usual  preflight 
briefings  until  compared  VRS  to  PATWAS  and  with  a rushed  FSS  Briefer." 

STILL  NEED  FSS  AFTER  VRS  BRIEFING.  "While  VRS  provides  specific  data  for  the 
airports  ect.,  FSS  is  better  for  broad  overview  of  general  meteorological 
conditions." 

"The  only  component  missing  is  that  describing  frontal  movements.  VRS  allows 
an  instantaneous  picture  ot  a specific  location,  but  one  must  call  FSS  to 
receive  a general  overview  of  frontal  dynamics.  I dial  VRS  for  station/enroute 
weather  and  then  call  FSS  for  a general  overview  and  to  file." 

"It  was  generally  helpful  in  deciding  when  to  go  to  the  airport,  but  further 
details  from  FSS  were  nearly  always  necessary." 

"VRS  is  excellent  for  an  overview  of  weather  conditions,  but  feel  that  a 
comprehensive  brief  by  FSS  is  desirable  with  flight  of  longer  durations." 

ADD  MORE  WEATHER,  PIREPS,  NOTAMS,  SIGMETS,  AIRMETS,  WINDS  ALOFT.  "Believe 
institution  of  system  with  "weather  synopsis,  radar  reports,  NOTAM/SIGMET, 
AIRMET,  area  forecast"  capability  would  relieve  FSS  facilities  of  being  over- 
worked, especially  those  on  the  Eastern  Seaboard  Megalopolis  area." 

"On  flight  from  Washington  D.C.  to  Akron,  four  of  five  station  surface  obser- 
vations do  not  provide  frontal  or  airmass  situation,  cloud  layers,  tops, 
turbulence,  severe  weather  warnings,  NOTAMS." 

"The  service  was  good  as  a basic  overview.  However,  on  long-distance  flights, 

I had  to  contact  FSS  to  obtain  frontal  information,  AIRMETS,  SIGMETS,  and 
PIREPS." 

"I  would  like  to  be  able  to  request  important  NOTAMS  (i.e.,  runway  closed)  and 
any  thunderst srom  activity  along  or  near  my  route  of  flight." 

ADD  SYNOPSIS  OF  WEATHER.  "it  might  be  helpful  to  provide  for  a synopsis  of 
weather,  location,  and  movement  of  fronts,  etc;  on  an  area-wide  basis.  This 
could  further  reduce  the  reliance  on  other  sources  of  information." 

"No  general  synopsis  concerning  enroute  airmass  or  frontal  systems.  This 
should  be  added  with  pilot  giving  general  routing  info  for  general  synoptic 
brief,  in  addition  to  station  sequence  reports." 


i 


66 


"A  general  weather  synopsis  for  a given  flight  area,  especially  for  forecasted 
weather,  would  be  useful." 

"The  thing  needed  to  make  the  system  more  complete  would  be  a general  descrip- 
tion of  major  weather  factors  such  as  fronts,  movements  of  fronts,  and  general 
outlook  for  forecast  area." 

SUGGESTED  MKCHANICA1.  AND/OR  FORMAT  CHANGES.  "I'd  recommend  a longer  pause  from 
the  last  keyed  input  to  the  first  VRS  response.  Those  of  us  using  telephones 
which  have  the  pushbuttons  on  the  earpiece  carrier  (e.g..  Bell  Trimline)  often 
miss  the  beginning  of  a VRS  transmission." 

"Suggest  that  a b-hour  forecast  be  issued  along  with  the  initial  current  hourly 
weather  obs.  Most  pilots  will  be  interested  in  the  forecast  along  with  current 
weather. " 


"The  format  is  somewhat  awkward,  and  the  system  a little  slow.  It  would  be 
better  if  you  could  get  the  specific  info  wanted  directly  instead  of  going 
through  the  whole  format,  regardless  of  the  "jump  ahead"  feature." 


"How  about  specifying  a certain  time  for  forecast  weather.  Takes  to  much 
listening  to  obtain  info  for  12  to  15  hours  in  advance." 

DIFFICULT  TO  USE /OPERATE . "I’ve  reread  the  instructions  a number  of  times  and 
repeatedly  tried  the  system.  I'm  still  unable  to  get  all  the  info  I want  out 
of  VRS.  I've  had  trouble  with  numbers,  i.e.,  altitudes,  hours." 


"Could  not  make  the  system  work  at  first,  instructions  too  lengthy.  Tried 
the  example  and  got  nothing.  I finally  got  it  to  work." 

"If  an  error  is  made  interrogating  the  system,  it  is  time  consumming  to  make 
the  correction." 


"Keypad/touchtone  alphanumeric  system  is  much  too  complicated  for  the  user; 
it  requires  (1)  learning,  (2)  concentration  in  use,  (3)  remembering  how  to 
use  it." 


COMPARISON  OF  COMMENTS. 


The  correlation  between  the  type  and  frequency  of  comments  received  on  item 
12,  from  the  initial  as  compared  to  the  first  followup,  was  measured  using 
Spearman's  rank  correlation  coefficient  as  defined  by 


1-1 

n(nJ  - 1) 


rs  denotes  the  Spearman  rank-correlation  coefficient, 

n = number  of  paired  observations  (x^,  y^) 
Dj  ” rank  of  (X^)  - rank  (yi)  “ Rl-Si 


where 


The  comment  category  and  frequency  and  the  ranks  that  they  received  are  shown 
in  table  35.  The  comments  on  the  second  followup  were  not  similarly  tested, 

Reference:  "Nonparametric  Statistical  Inference,"  by  J.D.  Gibbons 


TABLE  35.  RANK  COMPARISON 


Initial 

First  Followup 

Rank 

(Initial ) 

Rank 

First  Followup 

Commending  the  VRS 

116 

76 

i 

i 

Have  Noc  Flown  Recently 

No  Longer  Fly-Moved 

39 

46 

3 

Still  Need  FSS  after  VRS 

46 

16 

2 

5 

Add  More  Weather 

36 

ia 

4 

3 

Add  Synopsis 

34 

18 

5 

3 

Suggested  Mechanical  or 

Format  Changes 

22 

15 

6 

6 

No  Touchtone  Phone  Available 

11 

16 

11 

5 

Difficult  to  Use 

14 

17 

9 

4 

Want  More  Hours 

16 

12 

7 

7 

VRS  Did  Noc  Worn  Properly 

Not  Satisfied 

15 
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9 
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7 

9 

14 

8 

Add  Route  Forecast 

13 

9 

10 

8 

Prefer  Plain  Language  to 
Phonetics 

9 

* 

12 

10 

Speech  Rate  and  Quality 

8 

3 

13 

11 

Want  Toll  Free  Number 

9 

1 

12 

13 

Prefer  Local  Time  Instead 
of  GMT 

a 

1 

13 

13 

Have  VRS  Record  Aircraft 

ID  or  Pilot  Name 

4 

2 

14 

12 

Good  for  Go/No  Go  Decision 

•y 

2 

15 

12 
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since  they  were  not  as  numerous  as  the  comments  received  on  the  initial  and 
first  followup.  This  disparity  prevented  a basis  of  comparison. 

The  Spearman  rank-correlation  coefficient  assumes  values  from  -1  to  +1.  A 
value  of  rg  equal  to  1 denotes  perfect  correspondence  between  the  ranks  of  the 
two  variables.  A value  of  rs  equal  to  -1  denotes  that  the  ranks  of  one  variable 
are  in  exactly  the  opposite  order  as  the  ranks  of  the  other  variable.  A value 
of  rs  near  zero  indicates  that  the  two  variables  are  independent. 

For  a sample  size  of  18  and  a = .05,  the  critical  region  consists  of  all  values 
of  r.  > .399.  The  value  of  rs  is  0.86.  Since  0.86  > .399,  the  value  of  rs  is 
in  tne  critical  region:  therefore,  we  reject  Ho  and  accept  — that  the  two  sets 
of  ranks  are  positively  correlated. 

RANDOM  SAMPLE  OF  NONRESPONDEN  i'S . 

A telephone  followup  was  conducted  on  a random  sample  of  nonrespondents  to 
determine  whether  or  not  they  held  strong  opinions  about  the  VRS.  The  answer 
is  that  they  do  not  and  that  there  need  be  no  concern  about  latent  adverse 
opinion.  Six  hundred  telephone  calls  were  made  to  individuals  who  did  not 
respond  to  the  questionnaire.  Each  individual  called  was  selected  by  using 
computer-generated  random  numbers.  Of  the  600  nonrespondents  called,  212 
were  actually  contacted.  Figure  41  provides  assurance  that  no  pockets  of 
vociferous  but  previously  unexpressed  hostility  to  the  VRS  exist  among  the 
nonrespondents.  Ninety-six  of  the  212  nonrespondents  contacted  or  45  percent 
indicated  that  the  reason  they  did  not  respond  to  the  questionnaire  was  that 
they  either  have  not  flown  recently,  they  no  longer  fly,  or  they  have  moved. 

It  is  extremely  important  to  obtain  information  from  the  nonrespondents  to 
mailed  questionnaires.  Voluntary  response  to  mailed  questionnaires  is  likely 
to  contain  an  intrinsic  bias  due  to  the  likelihood  that  people  who  are  inter- 
ested enough  to  answer  a questionnaire  form  a special  subset  as  compared  to 
those  who  are  more  apathetic.  As  one  writer  put  it  (reference  5): 

"Voluntary  response  to  mailed  questionnaires  is  perhaps  the  most  common  method 
of  social-science  data  collection  encountered  by  statisticians,  and  perhaps 
also  the  worst....  More  generally,  one  should  realize  that  voluntary  response 
is  such  a pervasive  problem  that  it  may  be  expected  to  introduce  bias  into 
any  survey  using  it.  Whether  the  subject  is  political  preference  or  university 
parking  policies,  the  intensely  interested  subset  is  certain  to  differ  from 
the  more  apathetic  element  of  the  population." 

SUMMARY  OF  MULTIVARIATE  ANALYSIS,  SUBGROUP  RESPONSES. 

The  preceding  subsections  indicate  that,  as  a group,  the  responses  to  question 
6 of  the  questionnaire  were  quite  favorable  to  the  VRS.  It  is  also  desirable 
to  know  whether  or  not  large  subgroups  of  respondents  are  similarly  favorably 
disposed.  Important  response  differentials  would  point  the  way  to  further 
improvements  that  might  be  required  in  the  VRS  to  increase  its  acceptability 
to  identifiable  segments  of  the  user  population. 
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A main  effects  analysis  of  variance  model  was  used  in  the  multivariate  analysis 
of  question  6.  The  program  was  run  on  an  International  Business  Machine  (IBM) 
369/9020  computer  using  the  program  distributed  by  International  Education 
Services.  The  mathematical  foundations  are  presented  by  Finn  (reference  5). 

The  input  data  consisted  of  the  returns  from  the  initial,  first  followup,  and 
second  followup  mailing. 

The  four  main  effects  were  the  following  classification  variables  listed  at 
the  beginning  of  the  questionnaire: 

1.  Weather  Rating 

2.  Total  Flying  Time 

3.  Flying  Time  in  12  Months  Prior  to 
Completing  Questionnaire 

A.  Number  of  Times  VRS  Used 

Two  levels  of  each  effect  were  used  in  the  analysis.  The  threshold  values 
separating  the  levels  are  shown  in  table  36. 

Question  6 of  the  VRS  questionnaire  was  scored  in  terms  of  a Likert-type  scale. 
Guilford  (reference  6 p.  A59)  discusses  the  justification  for  and  utility  of 
handling  five-point  response  scales  in  this  form.  Weights  of  1 through  5 are 
assigned  the  categories  in  the  five-point  scale.  Although  there  may  become 
debate  on  the  point,  it  is  commonly  considered  that  this  method  yields  an  in- 
terval scale  to  which  statistical  operations  appropriate  to  such  a scale  can 
be  applied  (see,  for  example,  Stevens,  reference  7,  p.  25). 

The  results  of  the  analysis  of  question  6 are  presented  below.  The  scale 
values  shown  in  tables  37,  38,  39,  and  AO  and  figures  A2  though  A5  were 
derived  by  assigning  numerical  values  to  each  of  the  five  response  categories 
for  each  item  in  question  6 and  computing  averages  weighted  by  the  number  of 
respondents  in  each  category.  The  value  "3"  was  assigned  to  the  middle  category 
(uncertain)  with  descending  and  ascending  unit  values  employed  for  adjacent 
categories  depending  on  whether  the  question  is  negative  or  positive  toward 
VRS . Thus,  values  greater  then  "3"  signify  a favorable  reaction  to  the  VRS. 

It  should  be  noted  that  all  of  the  stimulus  items  are  significantly  larger 
than  the  indifference  value  of  "3.” 

Tables  Al,  A2,  A3,  and  AA  show  the  univariate  F ratio  and  the  probability  value 
for  each  stimulus  item.  Table  Al  indicates  a statistically  significant 
difference  (p  < .05)  between  the  VFR  and  IFR  pilot  with  respect  to  item  6c 
"Speech  too  fast."  The  IFR  pilot  responded  more  favorably  to  item  6c;  that  is  to 
say,  he  responded  more  frequently  with  Disagree  or  Strongly  Disagree  than  did  the 
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TABLE  36.  VALUES  ASSIGNED  TO  THE  CLASSIFICATION 

PARAMETERS  USED  IN  THE  MULTIVARIATE  ANALYSIS 


i 


Classification 

Parameter 

Analysis  Level  Number 

1 2 

1 

j 

1. 

License /Rating 

VFR 

(Student , 
Private  VFR, 
Commercial  VFR) 

IFR 

(Private  IFR, 
Commercial  IFR 
Air  Transport) 

2 m 

Total  Flying  Hours 

0 - 400 

> 400 

• • 

3. 

Annual  Flying  Hours 

0-50 

> 50 

4 • 

No.  Times  VRS  Used 

0-10 

> 10 

TABLE  37.  SCALE  VALUE  COMPARISONS  FOR  QUESTION  6,  VFR  VERSUS  IFR 


VFR  IFR 

Scale  Values 


a. 

Order  of  Information 

4.11 

4.20 

b. 

Information  Sufficient  for  Decision  to 
Airport 

4.08 

4.02 

c. 

Speech  Rate 

3.93 

4.13 

d. 

Clear  Mental  Picture  of  Weather 

3.67 

3.72 

e. 

Amount  of  Information  Provided 

3.43 

3.41 

f . 

VRS  Operating  Procedures 

3.91 

4.04 

g. 

Intelligibility 

3.91 

3.82 

h. 

Speech  Rhythm  and  Intonation 

3.85 

4 .00 

i. 

Pleasantness  of  VRS  Speech 

4.27 

4.35 

j • 

Briefing  Satisfaction 

4.05 

4.01 

Note:  Values  greater  than  3 signify  a favorable  reaction. 
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TABLE  38.  SCALE  VALUE  COMPARISONS  FOR  QUESTION  6,  400  HOURS  OR  UNDER 
VERSUS  OVER  400  HOURS  TOTAL  FLYING  TIME 


<_  400  > 400 

Scale  Values 


i 

a. 

Order  of  Information 

4.15 

4.17 

H 

b. 

Information  Sufficient  for  Decision  to 

4.02 

4.07 

Airport 

*1 

c. 

Speech  Rate 

3.93 

4.12 

M 

d. 

Clear  Mental  Picture  of  Weather 

3.67 

3.71 

N 

i 

e. 

Amount  of  Information  Provided 

3.41 

3.42 

f. 

VRS  Operating  Procedures 

3.99 

3.99 

g. 

Intelligibility 

3.85 

3.86 

h. 

Speech  Rhythm  and  Intonation 

3.82 

4.00 

i. 

Pleasantness  of  VRS  Speech 

4.29 

4.33 

j. 

Briefing  Satisfaction 

4.04 

4.02 

Note 

: Values  greater  than  3 signify  a favorable 

reaction. 

TABLE  39.  SCALE  VALUE  COMPARISONS  FOR  QUESTION  6,  50  HOURS  OR  UNDER  VERSUS 
OVER  50  HOURS  ANNUAL  FLYING  TIME 

_<  50  >50 

Scale  Values 


a. 

Order  of  Information 

4.13 

4.19 

b. 

Information  Sufficient  for  Decision  to 
Airport 

4.01 

4.08 

c. 

Speech  Rate 

3.90 

4.15 

d. 

Clear  Mental  Picture  of  Weather 

3.69 

3.70 

e. 

Amount  of  Information  Provided 

3.50 

3.35 

f. 

VRS  Operating  Procedures 

3.97 

3.99 

g* 

Intelligibility 

3.91 

3.81 

h. 

Speech  Rhythm  and  Intonation 

3.91 

3.94 

i. 

Pleasantness  of  VRS  Speech 

4.31 

4.31 

j- 

Briefing  Satisfaction 

4.06 

4.01 

Note:  Values  greater  than  3 signify  a favorable 

reaction. 

TABLE  40.  SCALE  VALUE  COMPARISONS  FOR  QUESTION  6,  VRS  USED  10  TIMES  OR 
UNDER  VERSUS  VRS  USED  OVER  10  TIMES 


£10  >10 
Scale  Values 


a. 

Order  of  Information 

4.11 

4.27 

b. 

Information  Sufficient  for  Decision  to 
Airport 

4.01 

4.12 

c. 

Speech  Rate 

3.93 

4.25 

d. 

Clear  Mental  Picture  of  Weather 

3.67 

3.74 

e. 

Amount  of  Information  Provided 

3.46 

3.35 

f. 

VRS  Operating  Procedures 

3.85 

4.23 

g. 

Intelligibility 

3.93 

3.73 

h. 

Speech  Rhythm  and  Intonation 

3.91 

3.98 

i. 

Pleasantness  of  VRS  Speech 

4.27 

4.39 

j. 

Briefing  Satisfaction 

3.97 

4.15 

Note 

::  Values  greater  than  3 signify  a favorable 

reaction. 

VFR  IFR 
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ORDER  OF  INFORMATION 

INFORMATION  SUFFICIENT  FOR  DECISION  TO  AIRPORT 
SPEECH  RATE 

CLEAR  MENTAL  PICTURE  OF  WX 
AMOUNT  OF  INFORMATION 
OPERATING  PROCEDURES 
INTELLIGIBILITY 
SPEECH  RHYTHM  AND  INTONATION 
PLEASANTNESS  OF  SPEECH 
BRIEFING  SATISFACTION 


1_ 


NUMBER  OF  TIMES  VRS  USED 
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FIGURE  45.  SCALE  VALUE  COMPARISONS  FOR  QUESTION  6,  USED 
VRS  OVER  VERSUS  UNDER  10  TIMES 


TABLE  41.  LEVELS  OF  SIGNIFICANCE  FOR  ITEMS  IN  QUESTION  6,  LICENSE/RATING 


a.  Order  of  Information  Satisfactory 

b.  Information  Sufficient  for  Decision  to 
Airport 

c.  Speech  Rate  Was  Too  Fast 

d.  Clear  Mental  Picture  of  Weather 

e.  Not  Enough  Information  Was  Provided 

f.  VRS  Operating  Procedures  Were  Difficult 

g.  The  Speech  Was  Intelligible 

h.  Speech  Rhythm  and  Intonation  Was 
Unnatural 

i.  Speech  on  the  VRS  Was  Unpleasant 

j.  The  Briefing  Was  Satisfactory 


UNIVARIATE  F 


P LESS  THAN 


SB 


i 

; • 


rv  » 


TABLE  42.  LEVELS  OF  SIGNIFICANCE  FOR  ITEMS  IN  QUESTION  6,  TOTAL  FLYING  TIME 


ITEM 

UNIVARIATE  F 

P LESS 

a. 

Order  of  Information  Satisfactory 

0.59 

0.44 

b. 

Information  Sufficient  for  Decision  to 
Airport 

2.42 

0.12 

c. 

Speech  Rate  Was  Too  Fast 

0.47 

0.49 

d. 

Clear  Mental  Picture  of  Weather 

0.04 

0.84 

e. 

Not  Enough  Information  Was  Provided 

0.07 

0.79 

f. 

VRS  Operating  Procedures  Were  Difficult 

0.95 

0.33 

g. 

The  Speech  Was  Intelligible 

0.69 

0.41 

h. 

Speech  Rhythm  and  Intonation  Was 
Unnatural 

1.44 

0.23 

i. 

Speech  on  the  VRS  Was  Unpleasant 

0.16 

0.69 

j. 

The  Briefing  Was  Satisfactory 

0.00 

0.95 

TABLE  43.  LEVELS  OF  SIGNIFICANCE  FOR  ITEMS  IN  QUESTION  6,  ANNUAL  FLYING  TIME 


ITEM 


a.  Order  of  Information  Satisfactory 

b.  Information  Sufficient  for  Decision  to 
Airport 

c.  Speech  Rate  Was  Too  Fast 

d.  Clear  Mental  Picture  of  Weather 

e.  Not  Enough  Information  Was  Provided 

f.  VRS  Operating  Procedures  Were  Difficult 

g.  The  Speech  Was  Intelligible 

h.  Speech  Rhythm  and  Intonation  Was 
Unnatural 

i.  Speech  on  the  VRS  Was  Unpleasant 

j.  The  Briefing  Was  Satisfactory 


UNVARIATE  F 

0.21 

1.13 

3.84 

0.00 

2.34 

0.01 

0.47 

0.16 

0.14 

0.24 


P LESS  THAN 

0.64 

0.29 

0.05 

1.00 

Q.13 

0.94 

0.49 

0.69 

0.71 

0.62 


i 


TABLE  44.  LEVELS  OF  SIGNIFICANCE  FOR  ITEMS  IN  QUESTION  6,  VRS  USAGE 


a.  The  Order  in  Which  the  Information 

Was  Presented  Was  Satisfactory. 

b.  The  Infromation  Was  Sufficient  to 
Make  a Decision  to  Go  to  the  Airport. 

c.  The  Speech  Rate  Was  Too  Fast. 

d.  The  Information  Provides  a Clear  Mental 
Picture  of  the  Weather. 

e.  Not  Enough  Information  Was  Provided 

f.  The  VRS  Operating  Procedures  Were 
Difficult. 

g.  The  Speech  Was  Intelligible. 

a.  The  Speech  Rhythm  and  Intonation  Was 
Unnatural. 

L.  The  Speech  on  the  VRS  Was  Unpleasant. 

J.  The  Briefing  Was  Satisfactory. 


UNIVARIATE  F 

P LESS  THAN 

4.19 

0.04 

1.72 

0.19 

8.71 

0.00 

0.54 

0.46 

0.58 

0.45 

12.23 

0.00 

1.67 

0.20 

0.80 

0.37 

3.07 

0.08 

5.66 

0.02 

VFR  pilot  (see  figure  42).  No  statistically  significant  difference  was 
found  between  pilots  with  less  than  or  equal  to  400  hours  of  flying  time  and 
pilots  with  more  than  400  hours.  Table  43  indicates  a statistically  significant 
difference  (p  i .05)  between  pilots  with  less  than  or  equal  to  50  hours  flown 
during  the  past  12  months  and  those  that  flew  more  than  50  hours  with  respect 
to  item  6c,  "Speech  too  fast."  The  pilot  with  more  than  50  hours  of  flight 
time  during  the  past  12  months  responded  more  favorably  to  item  6c  (by  disagree- 
ing or  strongly  disagreeing  with  the  statement) , thus  producing  a higher  scale 
value  (see  figure  44).  Numerous  statistically  significant  differences  were 
found  between  respondents  who  used  the  VRS  less  than  or  equal  to  10  times  and 
respondents  who  used  the  system  more  than  10  times  with  respect  to  the  following 


items 

6(a) 

Order  of  Information  Satisfactory 

6(c) 

Speech  Too  Fast 

? 6(f) 

Operating  Procedures  Difficult 

i 6 ( j) 

Briefing  Satisfactory 

In  all  of  the  above  cases,  pilots  who  used  the  system  more  than  10  times 
responded  more  favorably  to  the  VRS  than  pilots  who  used  the  system  10  times 
or  less.  That  is  to  say,  the  more  frequent  users  of  the  VRS  (>  10)  rated  the 
system  significantly  higher  in  the  above  listed  areas  than  the  less  frequent 
users  (<. 10)  (see  figure  45).  This  was  especially  true  for  item  6g,  "The  VRS 
operating  procedures  were  difficult."  For  this  item,  p was  less  than  0.0006. 

As  might  be  expected,  the  more  frequent  user  found  the  VRS  operating  procedures 
less  difficult. 


VRS  TELEPHONE  ACTIVITY 


The  following  explanation  and  description  of  the  procedures  used  to  collect 
VRS  telephone  activity  data  was  provided  by  the  Transportation  System  Center 
(TSC)  . 

VRS  STATISTICAL  REPORT. 

During  each  VRS  operating  day,  raw  statistical  data  are  stored  in  a file.  This 
file  is  the  basis  of  the  statistical  report.  The  first  block  of  the  file  con- 
tains both  the  starting  time  and  the  finish  time.  These  values  of  data  and 
minute  of  day  are  presented  on  the  first  page  of  the  report. 

The  remainder  of  the  raw  statistical  data  file  consists  of  records,  with  one 
record  per  event.  The  event  records  are  subdivided  by  channel.  Within  each 
channel  the  records  are  in  chronological  order.  The  records  of  each  channel 
are  divided  into  users.  A new  user  starts  with  a channel  ringing  event  and 
terminates  with  a channel  hangup  event. 
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Each  line  in  the  report  represents  data  for  single  user.  However,  no  user  is 
included  unless  he  did  more  than  dial  the  phone.  A daily  summary  is  included 
at  the  end  of  the  report. 

There  are  three  report  forms;  the  header,  the  user  report,  and  the  daily  summary. 
Each  will  be  described  below. 

THE  HEADER.  The  header  has  the  start  and  finsh  time  for  the  raw  data  file, 
with  the  following  information: 


1)  Start  Date 
Block 
Byte 

Time 

2)  Finish  Date 
Block 

Byte 

Time 


year,  month,  date 
first  data  block 
first  data  byte 

minutes  since  midnight  Greenwich  Mean  Time  (G.m.t.) 
year,  month,  day 
last  data  block 
last  data  byte 

minutes  since  midnight  G.m.t. 


USER  REPORT.  The  User  Report  has  one  line  per  user.  There  are  17  columns, 
divided  into  4 groups.  The  first  group  has  one  column,  G.m.t.  time.  This 
is  the  time  of  the  user  channel  hang  up. 

Group  two  has  four  columns,  each  of  which  is  a count  of  user  requests.  Column 
Loc  is  a count  of  location  indentifiers  entered.  Column  SA  is  a count  of  SA 
reports  requested.  Column  FT  is  a count  of  FT  reports  requested.  Column  FD 
is  a count  of  FD  reports  requested. 

Group  three  has  six  columns  and  shows  the  allocation  of  user  time.  Column 
Time  shows  the  total  time  allocated  to  the  user.  Column  Over  shows  the  time 
allocated  to  overhead.  Column  Loc  shows  the  time  allocated  to  entering 
location  identifiers.  This  is  from  entrance  of  the  first  letter  until  the 
message  is  sent  to  the  PDF  11-70  and  accepted  by  the  PDP  11-70.  Column  SA 
is  time  from  the  SA  request  until  all  SA's  have  been  spoken.  Column  FT  is  the 
time  from  a request  for  FT  reports  until  all  FT  reports  are  spoken.  Column 
FD  is  the  time  from  the  FD  request  until  all  the  FD  reports  are  spoken. 

Group  four  has  six  columns,  which  contain  the  counts  of  how  many  times  the 
user  used  the  special  functions. 
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Special  Functions 
Title 


Stop 
Go 
RPT 
DEL 
JMP 
BEG 

SUMMARY  REPORT . The  fourth  report  type  is  a summary  of  the  day's  activity. 
The  first  column  is  the  hour  G.m.t.  The  second  column  is  the  number  of 
users  during  the  hour.  The  third  column  is  the  average  time  used  by  each 
user.  The  fourth  column  is  the  maximum  number  of  simultaneous  users  during 
the  hour. 

Below  this  is  the  total  number  of  user.  This  is  after  deleting  users  who 
rang  up  and  then  hung  up  the  channel. 

The  average  time  is  the  total  time  by  all  users,  divided  by  the  number  of 
users.  Next  is  the  maximum  number  of  simultaneous  users  for  the  entire  day. 

The  fractional  percent  is  found  by  summing  the  times  for  all  users  and  divi- 
ding the  time  for  each  purpose  by  the  total  time  users  were  on  the  system. 


Function 

Stop 

Go 

Repeat 

Cancel  last  location  identifier 
Skip  remainder  of  report 
Recycle  to  beginning  protocol 


ANALYSIS  OF  THE  VRS  STATISTICAL  FILE. 

During  each  VRS  operating  day,  a trace  file  is  maintained.  In  block  zero  of 
the  file  is  the  starting  date  and  time  in  seconds  and  finishing  date  and  time 
in  seconds.  The  first  page  of  the  statistical  report  contains  these  data  as 
date  and  time  in  minutes  for  identification. 

The  remainder  of  the  file  contains  event  records  and  these  are  used  for 
analysis.  The  event  records  are  subdivided  into  groups  based  on  channel  number. 
The  events  records  are  in  chronological  order.  User's  records  are  defined  as 
all  those  records  for  a channel  which  are  between  a channel  ringing  record  and 
the  first  channel  hungup  record  inclusive.  Specifically,  the  channel  ringing 
is  when  the  lower  byte  of  US.FLC  equals  five  and  the  channel  hungup  is  when 
the  lower  byte  of  US.FLC  equals  six.  The  statistics  for  each  user  is  put  on 
one  line  of  the  report.  However,  any  user  whose  second  event  is  channel 
hungup  is  not  recorded  in  the  statistics. 

Using  the  user  data,  the  gross  statistics  are  computed.  The  time  of  day  in 
column  one  is  the  G.m.t.  time  on  the  event  record  channel  hungup.  The 
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briefing  time  in  column  six  is  the  difference  in  tine  between  the  time  of  day 
on  the  channel  ringing  event  and  the  channel  hungup  event. 

In  order  to  get  the  counts  in  columns  two  through  five,  each  user  records  are 
divided  into  passes.  A pass  consists  of  two  phases,  (1)  a location  identifier 
entering  phase  and  (2)  a report  requesting  phase. 

The  location  identifier  count  in  column  two  is  done  in  two  parts.  First,  the 
location  entering  phase  is  checked  to  locate  the  record  which  has  the  greatest 
location  identifer  count.  This  is  the  location  count  for  the  pass.  Then  the 
location  count  for  the  pass  is  summed  up  for  the  user  to  get  the  location 
count  for  the  user  in  column  two.  The  location  identifier  input  phase  is  de- 
fined by  the  high  order  byte  of  US. KEY  being  either  equal  to  one  (Loc  ID)  or 
five  (Nxt  Loc). 

The  report  count  is  computed  for  each  pass  by  checking  the  high  order  byte  of 
US. KEY.  If  a report  is  requested  during  that  pass  (13  for  SA's,  15  for  FT's, 

19  for  FD's),  the  number  of  reports  of  that  type  requested  fcr  that  pass  equals 
the  number  of  location  identifiers  entered  that  pass.  This  is  then  summed  over 
the  passes  for  that  user  and  entered  in  columns  three,  four,  and  five.  Use  of 
the  repeat  special  key  may  lead  to  getting  multiple  report  counts  in  one  pass. 

The  allocation  of  time  to  overhead,  location,  SA,  FT,  and  FD  are  allocated  by 
use  of  the  high  order  byte  of  US. KEY.  When  two  consecutive  records  for  one 
user  have  different  keyword  values,  the  time  between  them  is  allocated  to  the 
second  record  keyword  value.  For  this  purpose:  (1)  keyword  values  one  (Loc 
ID)  and  five  (Net  Loc)  are  considered  equal;  (2)  keyword  values  16  (wind) , 

17  (time),  and  18  (altitude)  are  FD  reports,  and  (3)  keyword  values  2 (brief) 
and  4 (brief  2)  are  assigned  to  the  report  request  preceding  them. 

An  example  of  assigning  keyword  values  brief  and  brief  2.  If  there  are  three 
consecutive  records  for  one  user  and  the  first  keyword  value  is  13  (SA) , the 
second  record  two  (brief),  and  third  record  four  (brief  2),  the  routine  assumes 
all  keyword  values  are  13  (SA)  . 

Columns  11  through  15  contain  counts  of  the  special  functions.  As  each  record 
is  processed,  the  low  order  byte  of  US.STA  is  checked.  If  the  byte  has  a 
value  greater  than  40  octal,  it  is  processed  by  incrementing  the  counter  for  that 
special  function. 

SPECIAL  FUNCTION  CODES 


FUNCTION 


OCTAL  VALUE 


Stop  42 
Go  43 
Begin  44 
Repeat  45 
Jump  46 
Delete  51 
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SUMMARY  OF  VRS  TELEPHONE  ACTIVITY. 

Table  45  shows  the  summary  of  the  VRS  telephone  activity  for  the  period 
June  1,  1978,  through  September  30,  1978.  (Data  collection  for  this  report 
ended  on  September  30,  1978). 


AOPA  SURVEY 


The  Aircraft  Owners  and  Pilots  Association  (AOPA),  at  their  own  expense,  prin- 
ted and  mailed  approximately  11,000  questionnaires  to  pilots,  requesting  their 
participation  in  evaluatitg  the  Voice  Response  System. 

The  data  received  from  that  survey,  gave  approximately  the  same  results  as 
those  contained  in  this  report. 
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CONCLUSIONS 


From  the  results,  it  is  concluded  that: 

1.  The  VRS  is  a practical  and  effective  means  hy  which  pilots  can}  in  many 
instances , obtain  a satisfactory  overview  of  weather  conditions.  For  preflight 

planning,  56  percent  of  the  pilots  indicated  that  it  was  unnecessary  to  call 
the  Flight  Service  Station  (FSS)  for  weather  information  after  receiving  the 
VRS  briefing.  Of  those  who  still  found  it  necessary  to  call  the  FSS  for 
weather  information  after  receiving  the  VRS  briefing,  90  percent  indicated  that 
the  VRS  briefing  reduced  the  time  on  line  with  the  FSS.  The  average  reduction 
in  time  on  line  with  FSS  for  preflight  planning  was  estimated  to  be  47  percent. 

2.  The  VRS  operating  procedures  are  simple  enough  for  most  pilots  to  master 
with  minimal  practice. 

3.  The  public  reaction  to  the  VRS  was  highly  favorable.  Analysis  of  the 
data  obtained  from  the  questionnaires  indicates  a high  level  of  acceptance  of 

the  VRS.  Ninety-three  percent  of  the  respondents  indicated  that  the  VRS  is 
either  "suitable  with  minor  changes"  or  "suitable  as  is." 

4.  No  significant  difference  in  the  results  existed  between  license  types 
or  experience  levels  of  general  aviation  pilots. 

5.  The  quality  of  speech  (i.e.,  pleasantness , rate  rhythm,  intonation,  and. 
intelligibility ) are  acceptable  foi • public  dissemination. 

6.  Additional  weather  products  and  the  provision  of  NOTAt-t's  would  provide 
for  a better  overview  of  weather  and  flying  conditions . 


RECOMMENDATIONS 


1.  The  following  seven  additional  products  should  be  incorporated  into  the 
Voice  Response  System  as  they  become  available. 

1.  PIREP's — Pilot  reports  on  weather 

2.  NOTAM's — Notices  to  airmen 

3.  SIGMET’s — Significant  weather  information 

4.  AIRMET's — Airmens  Meteorological  Information 

5.  Weather  Synopsis 

6.  Route  Forecast 

7.  Area  Forecast 

2.  The  spoken  name  readback  method  should  be  used  for  location  indentif iers. 

3.  The  impact  of  the  Voice  Response  System  on  FSS  specialist  workload  was 
not  and  cannot  be  determined  in  the  Washington  metropolitan  area  due  to  many 


Improvements  such  as  MAPS,  improved  PATWAS,  TEL-TWEB,  800  lines,  and  consoli- 
dation. In  the  event  a second  VRS  installation  is  implemented,  it  is  recommended 
that  a further  study  be  conducted  to  determine  the  effects  of  the  VRS  on  special- 
ists' workload  and  that  this  study  be  performed  in  a controlled  field  environ- 
ment. 
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VOICE  RESPONSE  SYSTEM 


System  Access  dial  202-347-3222 


PRODUCTS  AVAILABLE: 

• HOURlv  SURFACE  OBSERVATIONS 

• TERMINAL  FORECASTS 
•winrs  Ainrr 


FRONT 


DEPARTMENT  OF  TRANSPORTATION 

FEDERAL  AVIATION  ADMINISTRATION 


Form  Approved 

OMB  No.  04-S7P010 

Use  Expires  September  1978 


August  1978 


In  July  of  this  year,  we  sent  you  a questionnaire  soliciting  your 
opinion  of  the  Voice  Response  System  (VRS)  which  has  been  demon- 
strated in  the  Washington,  D.  C. , and  surrounding  area.  If  you 
have  already  responded  to  the  survey  questions,  please  disregard 
this  reminder  and  accept  our  sincere  appreciation  for  your 
cooperation.  If  you  have  not  yet  completed  the  questionnaire,  or 
if  you  were  not  able  to  access  the  VRS  prior  to  the  time  you  received 
the  first  questionnaire  and  have  now  had  an  opportunity  to  do  so,  we 
earnestly  request  that  you  complete  the  questionnaire  as  soon  as 
possible. 


The  success  of  this  survey  depends,  to  a large  extent,  on  the 
number  of  responses  obtained.  Therefore,  we  are  most  anxious  to 
get  your  reaction  to  the  VRS  concept.  Your  voluntary  participation 
will  help  ensure  that  the  flying  public  gets  the  best  available 
weather  information  which,  in  turn,  will  help  make  flying  safer  and 
more  enjoyable.  Since  we  do  have  a time  limit  on  this  survey,  we 
would  appreciate  its  return  within  fourteen  (14)  days. 


Thank  you  for  your  cooperation 


DAVID  J.  ^USSTteL  v 

Director,  Sy^tjms  Research 
and  Development  Service 


FIGURE  A-2.  FIRST  FOLLOWUP  COVER  LETTER 


VOICE  RESPONSE  SYSTEM 


System  Access  dial  202-347-3222 


FRONT 


PRODUCTS  AVAILABLE: 

• HOURl*  SURFACE  OBSERVATIONS 
e TERMINAL  FORECASTS 
0 WINDS  ALOFT 

LETTERS  lOf  NTIFV  APPROPRIATE  KEY 

SELECT  LETTER  POSTHON  ON  KEY  SV  1.  2 OR  S 
NUMBERS  ENTER  DIO  ITS  ONLY 


BACK 


FIGURE  A-l.  VRS  WALLET-SIZE  CARD 


DEPARTMENT  OF  TRANSPORTATION 

FEDERAL  AVIATION  ADMINISTRATION 


Form  Approved 

OMB  No.  04-S78010 

Use  Expires  September  1978 


August  1978 


SUBJECT:  Voice  Response  System  (VRS)  Survey 


In  early  August  of  this  year,  we  sent  you  a follow-up  letter  and 
questionnaire  soliciting  your  views  and  opinions  on  the  Voice 
Response  System  (VRS) , which  has  been  demonstrated  in  Washington, 

D.  C. , and  surrounding  areas.  If  you  have  not  had  a chance  to  fill 
out  the  questionnaire,  we  earnestly  request  that  you  take  a few 
minutes  to  complete  the  form.  In  the  event  that  you  may  have 
misplaced  the  ques cionnaire , we  are  enclosing  another  copy.  This 
voluntary  questionnaire  is  preaddressed  and  self  mailing.  Since 
we  do  have  a time  limit  on  the  survey,  we  would  appreciate  its 
return  within  14  days. 


In  a survey  of  this  kind,  meaningful  results  depend,  in  a large 
measure,  on  the  magnitude  of  participation.  The  ultimate 
objective  of  the  VRS  test  is  to  help  make  flying  safer  and  more 
enjoyable  for  you  and  other  members  of  the  flying  public. 


We  would  be  most  grateful  for  your  help  in  accomplishing  this 
task. 


Thank  you  for  your  cooperation 


DAVID  J.  StfHFTEL 
Director , Systems  Research 
and  Development  Service 


Enclosure 


FIGURE  A-4 , SECOND  FOLLOWUP  COVER  LETTER 
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DEPARTMENT  OF  TRANSPORTATION 

FEDERAL  AVIATION  ADMINISTRATION 

WASHINGTON,  O.C.  20991 


August  1978 


TO:  All  Computer  Generated  Voice  Users 

SUBJECT:  Pilot  Evaluation  of  the  Voice  Response  System 

( VRS) 


For  the  past  4 months,  the  Federal  Aviation  Administration 
(FAA)  has  been  conducting  a public  demonstration  of  the 
experimental  computer  generated  VRS.  Pilot  participation 
thus  far  in  the  demonstration  has  been  excellent  and 
interest  in  the  concept  from  the  general  public  and 
technical  communities  has  been  abundant.  The  performance 
of  the  system  has  been  improved  since  the  start  of  the 
demonstration  in  April  providing  increased  system 
availability  to  pilots. 

In  July,  a questionnaire  was  mailed  by  the  FAA  to  a 
scientifically  selected  sample  of  pilots  to  obtain  user 
reaction  and  input  on  the  VRS  concept.  Additional 
questionnaire  forms  have  also  been  made  available  by  user 
organizations  in  the  Washington,  D.  C.,  area.  We  encourage 
pilot  participation  in  these  surveys  and  particularly 
solicit  the  input  of  those  pilots  selected  in  the  FAA's 
sample.  To  assure  all  pilots  have  an  opportunity  to 
voluntarily  take  part  in  the  VRS  survey,  the  FAA  will 
distribute  post-paid  questionnaire  forms  to  fixed-based 
operators  facilities  in  the  Washington,  D.  C.,  area.  These 
forms  will  be  made  available  through  September  30,  1978,  at 
which  time  all  data  collection  will  be  terminated. 


In  response  to  many  inquiries  regarding  information  on 
location  identifiers  for  system  use,  several  reference 
charts  have  been  assembled  and  are  enclosed  for  your 
convenience.  We  believe  this  data  will  greatly  enhance 
your  ability  to  effectively  use  the  system. 

Again,  we  encourage  your  participation  in  the  evaluation 
survey  and  take  this  opportunity  to  thank  you  for  your 
interest  and  support  during  this  important  phase  of  VRS 
development . 


Sincerely , 


DAVID1^.  SHE\’TEL 
Di  rector*", -Systems 
and  Development 


Research 

Service 


Enclosure 


FIGURE  A- 5.  LETTER  TO  PILOTS  NOT  SELECTED  FOR  VRS  SURVEY 
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